Overview
Sparse retrieval represents documents and queries as high-dimensional sparse vectors where most elements are zero. Traditional methods like BM25 and TF-IDF create sparse representations based on term frequencies, while modern neural approaches like SPLADE learn sparse vectors.
Traditional Sparse Methods
BM25 (Best Matching 25)
- Most popular sparse ranking function
- Considers term frequency and document length
- Probabilistic retrieval framework
- Default in Elasticsearch and most search engines
- Formula accounts for:
- Term frequency in document
- Inverse document frequency
- Document length normalization
TF-IDF
- Term Frequency × Inverse Document Frequency
- Simpler than BM25
- Still widely used
- Foundation for many IR systems
Neural Sparse Retrieval
SPLADE (Sparse Lexical And Expansion)
- Learns sparse vectors with neural networks
- Term weighting: adjusts importance of words
- Term expansion: adds related terms not in text
- Compatible with inverted indexes
- Performance competitive with dense methods
Advantages of Neural Sparse
- Semantic understanding via term expansion
- Interpretable (can see which terms matter)
- Works with existing inverted index infrastructure
- Faster than dense in some scenarios
Sparse Vector Characteristics
Dimensionality
- Size = vocabulary size (10K-100K+)
- Most values are zero (sparse)
- Efficient storage with sparse formats
Inverted Index
- Maps terms to document IDs
- Only stores non-zero entries
- Enables fast keyword lookup
- Fundamental to search engines
Advantages
- Exact Matching: Finds specific keywords, IDs, names
- Interpretable: Can see which terms matched
- Fast: Optimized inverted index lookups
- No Embedding Required: Works directly with text
- Established: Decades of optimization
Limitations
- Vocabulary Mismatch: Misses synonyms, paraphrases
- No Semantics: Doesn't understand meaning
- Language Specific: Each language needs separate handling
- Rare Terms: Struggles with low-frequency words
Use Cases
- Exact Keyword Search: Product codes, IDs, names
- Legal/Medical: Specific terminology matters
- Code Search: Function/variable names
- Hybrid Search: Combined with dense retrieval
- Filtering: Narrow down before dense reranking
Hybrid Sparse + Dense
Combining both approaches:
- BM25 for exact keyword matching
- Dense vectors for semantic similarity
- Reciprocal Rank Fusion to merge
- Best overall retrieval quality
Implementation
Traditional Sparse
- Elasticsearch (BM25)
- Apache Solr (TF-IDF, BM25)
- Whoosh (Python)
- Tantivy (Rust)
Neural Sparse
- SPLADE models
- Pinecone (sparse-dense hybrid)
- OpenSearch (hybrid search)
- Custom implementations
- Index compression
- Term pruning
- Early termination
- Caching frequent queries
- Distributed inverted indexes
Pricing
Varies by search engine; Elasticsearch/OpenSearch pricing applies.
