text2vec

// Overview

text2vec converts text — words, sentences or paragraphs — into vectors, and computes similarity between them. It is a single Python package that implements several representation methods side by side: Word2Vec, RankBM25, BERT, Sentence-BERT, CoSENT and BGE, each with both inference and training code, so you can compare approaches on the same semantic-matching task rather than committing to one upfront.

Its centre of gravity is Chinese text. The project trains and publishes its own matching models on Hugging Face — `text2vec-base-chinese` (a CoSENT model trained on Chinese STS data), `text2vec-base-chinese-sentence` and `text2vec-base-chinese-paraphrase` for sentence-to-paragraph matching, `text2vec-bge-large-chinese` supervised with CoSENT on top of BAAI/bge-large-zh-noinstruct, and `text2vec-base-multilingual` for cross-language matching including English.

The library's own contribution on the modelling side is CoSENT (Cosine Sentence), a ranking-style loss that brings training closer to how the model is actually used at prediction time — you take the sentence vector and compare cosines — which the project reports converges faster and scores better than Sentence-BERT on its evaluation sets. Every published model can be fine-tuned further on your own data. Version 1.2.9 added multi-process inference across several GPUs or CPUs and a command-line tool for batch vectorising text from a script.

// What it does

• One API over Word2Vec, RankBM25, Sentence-BERT, CoSENT and BGE representations
• Pretrained Chinese and multilingual matching models published on the Hugging Face Hub
• CoSENT ranking loss with training code, plus fine-tuning support on every published model
• Multi-GPU and multi-CPU inference via multi-process execution (v1.2.9+)
• Command-line tool for batch text vectorisation from a script
• Published comparisons of each method's effectiveness on semantic-matching benchmarks

// Getting started

text2vec is on PyPI and needs PyTorch. Install torch first, then the package.

pip install torch  # or: conda install pytorch
pip install -U text2vec

from text2vec import SentenceModel

m = SentenceModel()
m.encode("如何更换花呗绑定银行卡")
# Embedding shape: (768,)

from text2vec import SentenceModel, Word2Vec

t2v_model = SentenceModel("shibing624/text2vec-base-chinese")
sbert_model = SentenceModel("shibing624/text2vec-base-multilingual")
w2v_model = Word2Vec("w2v-light-tencent-chinese")

sentences = [
    "银行卡",
    "如何更换花呗绑定银行卡",
    "This framework generates embeddings for each input sentence",
]
sentence_embeddings = model.encode(sentences)
print(type(sentence_embeddings), sentence_embeddings.shape)
# <class 'numpy.ndarray'> (3, 768)

git clone https://github.com/shibing624/text2vec.git
cd text2vec
pip install -r requirements.txt
pip install --no-deps .

Commands and code are distilled from the project's own documentation — always check the official repo for the latest.

// When to use it

• Embed Chinese documents for a RAG index where English-first embedding models underperform
• Build a semantic search or deduplication pipeline over a Chinese or mixed-language corpus
• Compare Word2Vec, BM25, Sentence-BERT and CoSENT on your own matching data before choosing one
• Fine-tune a published Chinese matching model on in-domain pairs with the included training code
• Batch-vectorise a large text dump from the command line across several GPUs

// How text2vec compares

text2vec alongside other open-source embedding models & inference tools AI/TLDR tracks, ranked by GitHub stars.