テキスト埋め込みと意味検索
文をベクトルにし、意味が近いものを取る
本ページの本文は英語で提供されています。タイトルと導入は日本語化されています。
この能力とは何か
Encodes a piece of text into a fixed-length vector so that semantically similar texts sit close together, then retrieves related items by nearest-neighbour lookup. Output is a ranked result list (item ids and scores), not prose. Unlike question answering it only finds possibly relevant material; it does not compose the answer.
技術的にどう実現するか
The mainstream design is a two-tower model: queries and documents are compressed by encoders, and training pushes true pairs above random negatives, typically with contrastive learning and large batches. At query time all document vectors are pre-computed into an index and an approximate nearest-neighbour search returns the top-k in milliseconds. Sparse and dense representations are often mixed to combine keyword hits with semantic recall.
代表的な製品
6Pinecone
2019マネージドなベクトルDBと類似検索
Hugging Face Hub
2016オープンモデルとデータセットの集積地
Transformers
2018単一のAPIで学習済みモデルを読み込み・学習
Together API
2022オープンモデル向けの推論API
Replicate
2019API経由でコミュニティのモデルを実行
Perplexity
2022検索しながら答え、どの答えにも出典が付く
関連する組織
代表的な用途
- Search over enterprise documents and code
- The recall stage of RAG pipelines
- Deduplication, clustering and topic discovery
- Recommendation and similar-content entry points
どう評価するか
- Recall@k
- Whether the top-k contain all relevant documents
- nDCG
- Discounted cumulative gain that accounts for rank position
- MRR
- Mean reciprocal rank of the first relevant result
限界と難しさ
- Semantic similarity is not relevance: near neighbours may merely share wording
- Cross-domain or cross-lingual use degrades noticeably without adaptation
- Chunking long documents severs context, and answers at chunk boundaries are easily missed