Overview
Galileo is an evaluation and observability platform designed for build and optimizing RAG (Retrieval-Augmented Generation) systems. It provides tools to evaluate various components of a RAG pipeline including embedding models, rerankers, retrievers, and LLM outputs.
Features
- Embedding Model Evaluation: Compare and benchmark different embedding models (e.g., OpenAI text-embedding-3-small vs. HuggingFace models) using retrieval metrics
- Reranker Evaluation: Test different reranking configurations (emb_k, rerank_k) to measure improvements in attribution and context adherence
- Attribution Analysis: Trace which retrieved chunks are attributed to generated answers, enabling granular error analysis
- Context Adherence Scoring: Measure reduction in hallucinations by evaluating how well generated answers stick to provided context
- Pipeline Observability: Comprehensive monitoring across the entire RAG pipeline, from retrieval through generation
- Sweep Experiments: Run parameter sweeps across embedding models, dimensions, LLM models, and retrieval/reranking parameters
- Individual Run Inspection: Analyze individual runs to identify which chunks were attributed and their attribution scores
Evaluation Metrics
- Attribution: Measures how relevant retrieved chunks are for addressing the question (e.g., 10% improvement observed with Cohere reranker)
- Context Adherence: Tracks reduction in hallucinations by measuring adherence to provided context (e.g., 5% improvement with reranking)
- Retrieval Metrics: Standard IR metrics like NDCG for evaluating reranker effectiveness
RAG Pipeline Components Supported
- Vector store integration (Pinecone and others)
- Contextual compression retrievers
- Language model integration (OpenAI GPT-3.5, GPT-4, etc.)
- Embedding model comparison
- Reranker evaluation (Cohere, custom rerankers)
Pricing
Free tier available to get started.
