# text2vec

> Turn Chinese or multilingual text into sentence embeddings, with training included

- Category: Retrieval & Knowledge (RAG) / Embedding Models & Inference
- Language: Python
- Licence: Apache-2.0
- Repository: https://github.com/shibing624/text2vec
- Website: https://pypi.org/project/text2vec/
- Canonical page: https://ai-tldr.dev/tools/text2vec/

## In short

text2vec converts text — words, sentences or paragraphs — into vectors, and computes similarity between them. It sits in the Embedding Models & Inference category of the Retrieval & Knowledge (RAG) tools and is open-source, written in Python and licensed Apache-2.0. To get started: install.

## Overview

text2vec converts text — words, sentences or paragraphs — into vectors, and computes similarity between them. It is a single Python package that implements several representation methods side by side: Word2Vec, RankBM25, BERT, Sentence-BERT, CoSENT and BGE, each with both inference and training code, so you can compare approaches on the same semantic-matching task rather than committing to one upfront.

Its centre of gravity is Chinese text. The project trains and publishes its own matching models on Hugging Face — `text2vec-base-chinese` (a CoSENT model trained on Chinese STS data), `text2vec-base-chinese-sentence` and `text2vec-base-chinese-paraphrase` for sentence-to-paragraph matching, `text2vec-bge-large-chinese` supervised with CoSENT on top of BAAI/bge-large-zh-noinstruct, and `text2vec-base-multilingual` for cross-language matching including English.

The library's own contribution on the modelling side is CoSENT (Cosine Sentence), a ranking-style loss that brings training closer to how the model is actually used at prediction time — you take the sentence vector and compare cosines — which the project reports converges faster and scores better than Sentence-BERT on its evaluation sets. Every published model can be fine-tuned further on your own data. Version 1.2.9 added multi-process inference across several GPUs or CPUs and a command-line tool for batch vectorising text from a script.

## What it does

- One API over Word2Vec, RankBM25, Sentence-BERT, CoSENT and BGE representations
- Pretrained Chinese and multilingual matching models published on the Hugging Face Hub
- CoSENT ranking loss with training code, plus fine-tuning support on every published model
- Multi-GPU and multi-CPU inference via multi-process execution (v1.2.9+)
- Command-line tool for batch text vectorisation from a script
- Published comparisons of each method's effectiveness on semantic-matching benchmarks

## Getting started

text2vec is on PyPI and needs PyTorch. Install torch first, then the package.

1. Install

   Install PyTorch for your platform, then text2vec.

```bash
pip install torch  # or: conda install pytorch
pip install -U text2vec
```

2. Encode a sentence

   SentenceModel with no arguments loads the default model and returns a NumPy array — a 768-dimensional vector per sentence.

```python
from text2vec import SentenceModel

m = SentenceModel()
m.encode("如何更换花呗绑定银行卡")
# Embedding shape: (768,)
```

3. Pick the model that matches your task

   Pass a Hugging Face model id to SentenceModel. Use the Chinese CoSENT model for Chinese semantic matching, the multilingual one for mixed Chinese and English, and Word2Vec for literal matching or a cold start.

```python
from text2vec import SentenceModel, Word2Vec

t2v_model = SentenceModel("shibing624/text2vec-base-chinese")
sbert_model = SentenceModel("shibing624/text2vec-base-multilingual")
w2v_model = Word2Vec("w2v-light-tencent-chinese")
```

4. Embed a batch

   encode also accepts a list and returns one row per input sentence.

```python
sentences = [
    "银行卡",
    "如何更换花呗绑定银行卡",
    "This framework generates embeddings for each input sentence",
]
sentence_embeddings = model.encode(sentences)
print(type(sentence_embeddings), sentence_embeddings.shape)
# <class 'numpy.ndarray'> (3, 768)
```

5. Install from source instead

   Clone the repository if you want to train or modify the models.

```bash
git clone https://github.com/shibing624/text2vec.git
cd text2vec
pip install -r requirements.txt
pip install --no-deps .
```

## When to use it

- Embed Chinese documents for a RAG index where English-first embedding models underperform
- Build a semantic search or deduplication pipeline over a Chinese or mixed-language corpus
- Compare Word2Vec, BM25, Sentence-BERT and CoSENT on your own matching data before choosing one
- Fine-tune a published Chinese matching model on in-domain pairs with the included training code
- Batch-vectorise a large text dump from the command line across several GPUs