# BEIR

> BEIR (Benchmarking IR) is a benchmark suite for evaluating information retrieval and vector search systems across multiple tasks and datasets. Useful for comparing vector database performance.

- **URL:** https://github.com/beir-cellar/beir
- **Category:** Benchmarks & Evaluation
- **Tags:** benchmark, evaluation, Vector Search, datasets
- **Updated:** 2026-04-22 19:20
- **Canonical page:** https://vectordb.ever.works/pt/items/beir

## Details

# BEIR

[BEIR](https://github.com/beir-cellar/beir) (Benchmarking IR) is a heterogeneous benchmark suite designed for evaluating information retrieval and vector search systems across a wide range of tasks and datasets. It provides a standardized framework for comparing the performance of NLP-based retrieval models and vector databases.

## Features
- **Heterogeneous Benchmark**: Includes 15+ diverse IR (Information Retrieval) datasets covering different domains and tasks.
- **Unified Evaluation Framework**: Offers a consistent and easy-to-use interface for evaluating retrieval models across all included datasets.
- **Dataset Variety**: Datasets span various domains such as web search, question answering, fact checking, financial QA, biomedical, news, and more. Notable datasets include MSMARCO, TREC-COVID, BioASQ, NQ, HotpotQA, FiQA-2018, Quora, DBPedia, FEVER, SciFact, and others.
- **Ready-to-Use Datasets**: Most datasets are publicly available and can be downloaded and used directly; some datasets require reproduction due to licensing.
- **Model and Dataset Integration**: Integrates with Hugging Face for models and datasets, facilitating easy experimentation.
- **Leaderboard**: Maintains a public leaderboard for performance comparison via Eval AI.
- **Extensive Documentation**: Provides a wiki with quick start guides, dataset details, metrics, and tutorials.
- **Python Support**: Installable via pip, compatible with Python 3.9+.
- **Community Collaboration**: Open to contributions and dataset/model submissions from the community.

## Pricing
- BEIR is an open-source project and is free to use.

## Links
- [GitHub Repository](https://github.com/beir-cellar/beir)
- [Wiki Documentation](https://github.com/beir-cellar/beir/wiki)
- [Hugging Face Models & Datasets](https://huggingface.co/BeIR)
- [Leaderboard on Eval AI](https://eval.ai/web/challenges/challenge-page/1897)

## Category
- benchmarks-evaluation

## Tags
benchmark, evaluation, vector-search, datasets

---

_This Markdown mirror is generated by Ever Works for AI agents. The canonical HTML page is at https://vectordb.ever.works/pt/items/beir._