Design a vector search stack: embeddings, ANN, hybrid retrieval, re-ranking, and safety.
- User
Design a vector search stack: embeddings, ANN, hybrid retrieval, re-ranking, and safety.
- Assistant
Vector Search Stack Design
This design outlines a production-ready vector search stack for semantic search applications (e.g., RAG for LLMs, recommendation systems). It covers embeddings, ANN (Approximate Nearest Neighbors), hybrid retrieval (vector + keyword), re-ranking, and safety. The stack prioritizes:
- Performance: Sub-100ms latency at scale (1M+ docs).
- Accuracy: Hybrid + rerank boosts MAP@10 by 20-30%.
- Cost: Open-source core with managed options.
- Scalability: Horizontal scaling via sharding/replication.
High-Level Architecture
[Data Ingestion] → [Embeddings] → [ANN Index + BM25] → [Hybrid Query] ↓ [Retrieval (Top-K)] ↓ [Re-ranking (Top-100→10)] ↓ [Safety Filters] ↓ [Results (with scores/metadata)]Tech Stack Summary:
Component Primary Tool Alternatives Why? Embeddings Hugging Face bge-large-en-v1.5OpenAI text-embedding-3-large, CohereOpen-source, SOTA on MTEB leaderboard (64.23 score). ANN Qdrant (vector DB) FAISS (local), Pinecone (managed), Milvus Native hybrid support, Rust-based (fast), easy scaling. Hybrid Qdrant Hybrid (HNSW + BM25) Weaviate, Elasticsearch + FAISS Fuse scores natively (alpha-blend). Re-ranking Cohere Rerank 3.0 BGE-reranker-large (HF), cross-encoder/ms-marco-MiniLM 5-10% NDCG lift, API or local. Safety HF unitary/toxic-bert+ OpenAI ModerationPerspective API, LlamaGuard Multi-layer (toxicity, PII, jailbreak). Assumptions: Text corpus (chunked to 512 tokens). Extendable to multimodal (e.g., CLIP for images).
1. Embeddings Generation
Convert text chunks to dense vectors (e.g., 1024-dim).
- Model:
BAAI/bge-large-en-v1.5(HF Transformers).- Normalize vectors:
vector /= np.linalg.norm(vector). - Prefix queries with "Represent this sentence for searching...:" for asymmetry.
- Normalize vectors:
- Ingestion Pipeline (Python + LangChain/Airflow):
from sentence_transformers import SentenceTransformer model = SentenceTransformer('BAAI/bge-large-en-v1.5') embeddings = model.encode(docs, batch_size=128, normalize_embeddings=True) - Storage: Upsert vectors + metadata (e.g., doc_id, chunk_id, text) to Qdrant.
- Best Practices:
- Chunking: RecursiveCharacterTextSplitter (500 tokens, overlap 20%).
- Dim. Reduction: PCA to 384 if needed (tradeoff: 5% accuracy loss).
- Cost: Free (local GPU), ~$0.0001/query via HF Inference Endpoints.
2. ANN Index
Fast approximate kNN search.
- Index Type: Qdrant with HNSW (Hierarchical Navigable Small World).
- Params:
m=16, ef_construct=100, ef=128(balance speed/accuracy). - Quantization: Scalar (8-bit) for 4x memory savings, <1% recall drop.
- Params:
- Setup (Docker or cloud):
# docker-compose.yml services: qdrant: image: qdrant/qdrant:latest ports: [6333:6333]from qdrant_client import QdrantClient client = QdrantClient("localhost:6333") client.create_collection("docs", vectors_config={"size":1024, "distance": "Cosine"}) client.upsert("docs", points=[{"id":i, "vector":emb, "payload":meta} for i,emb,meta in data]) - Query:
client.search("docs", query_vector=emb, limit=100)→ ~10ms @ 10M vectors.
3. Hybrid Retrieval
Combine semantic (vector) + lexical (BM25) search.
- Fusion: Qdrant native hybrid:
alpha=0.7(vector weight).query_emb = model.encode(query) hits = client.query_points( "docs", query=query_emb, query_filter=None, # Add metadata filters limit=100, search_params={"hnsw_ef":128, "exact":False}, bm25={"query":query}, # Keyword boost score=ScoreConfig(fuse_factor=0.7) # Hybrid score ) - Scoring: Normalized fusion:
final_score = α * cos_sim + (1-α) * bm25. - Benefits: Handles spelling errors, proper nouns (e.g., "Sherlock Holmes" → exact match).
4. Re-ranking
Refine top-100 to top-10 using query-document pairs.
- Model: Cohere Rerank v3 (API, relevance scores 0-1).
import cohere co = cohere.Client("YOUR_API_KEY") results = co.rerank(query=query, documents=[hit.payload['text'] for hit in hits], top_n=10) reranked = [{"text": res.document['text'], "score": res.relevance_score} for res in results.results] - Local Alt: HF
BAAI/bge-reranker-large(cross-encoder). - Latency: 50ms for 100 docs. Run on GPU for batching.
5. Safety Layer
Multi-stage guardrails post-retrieval.
- Pipeline:
- Toxicity Filter: HF
unitary/toxic-bert→ Drop if score > 0.5.from transformers import pipeline toxicity = pipeline("text-classification", "unitary/toxic-bert") safe_docs = [d for d in reranked if toxicity(d['text'])[0]['score'] < 0.5] - PII Detection: Presidio Analyzer → Redact/anonymize names, emails.
- Moderation: OpenAI API → Block hate/violence (categories: hate, sexual, etc.).
- Jailbreak/NSFW: Custom regex + LlamaGuard-2-8B → Flag prompt injection.
- Toxicity Filter: HF
- Config: Thresholds tunable; log blocked items for auditing.
- Edge Cases: Multilingual (use mDeBERTa), images (CLIP + safety classifiers).
Integration & Deployment
- Full Query Function:
def search(query: str, k: int = 5) -> list: emb = model.encode(query) hits = client.query_points(...) # Hybrid reranked = co.rerank(...) # Re-rank safe = safety_filter(reranked) # Safety return safe[:k] - Scaling:
Scale Setup Cost (monthly) <1M docs Local Qdrant + FAISS Free 10M+ Qdrant Cloud (3 pods) $200-500 1B+ Kubernetes + Ray $5k+ - Monitoring: Prometheus for latency/recall, A/B test rerankers.
- Eval Metrics: NDCG@10, MRR, Recall@100. Use BEIR/TREC datasets.
This stack achieves 95%+ recall with <100ms p99 latency. Customize for domain (e.g., legal: add legal-specific embeddings). Questions? Let's iterate! 🚀
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.