RAG Production Readiness Checklist

Data & Ingestion

• Document parsing tested on all formats
• Chunking strategy optimized
• Metadata extraction working
• Deduplication implemented
• Update/delete mechanisms in place
• Incremental ingestion tested
• Data quality validation
• Error handling for bad documents

Embeddings

• Model selected and justified
• Embedding dimensions optimized
• Batch processing efficient
• Error handling for embedding failures
• Retry logic implemented
• Rate limiting configured
• Cost monitoring in place

Vector Database

• Index type chosen and tested
• Query performance acceptable (p95 < target)
• Scaling strategy defined
• Backups automated
• Monitoring configured
• Connection pooling implemented
• Failover tested
• Capacity planning done

Retrieval

• Top-K value optimized
• Retrieval metrics (recall@K) measured
• Hybrid search configured (if needed)
• Reranking implemented (if beneficial)
• Filtering tested
• Query expansion evaluated
• Caching strategy defined
• Timeout handling implemented

LLM Integration

• Model selected (cost/quality balance)
• Prompt template optimized
• Context window managed
• Fallback models configured
• Rate limiting handled
• Cost tracking implemented
• Response streaming (if needed)
• Error handling robust

Evaluation

• Test set created
• Baseline metrics established
• Retrieval quality measured
• Generation quality assessed
• End-to-end performance tested
• Edge cases identified
• Regression tests automated
• A/B testing capability

Monitoring & Observability

• Request tracing implemented
• Latency tracking (p50, p95, p99)
• Error rate monitoring
• Cost tracking per request
• Token usage monitored
• Cache hit rates tracked
• Alerts configured
• Dashboards created

Security & Privacy

• Data access controls implemented
• PII detection/redaction
• Audit logging enabled
• API authentication required
• Rate limiting per user/key
• Input validation
• Output sanitization
• Compliance requirements met

Reliability

• Circuit breakers implemented
• Retry logic with backoff
• Graceful degradation
• Health checks configured
• Load testing completed
• Disaster recovery plan
• SLA/SLO defined
• On-call rotation established

User Experience

• Response time acceptable
• Source attribution working
• Confidence scores shown (if applicable)
• Error messages user-friendly
• Feedback mechanism implemented
• Query suggestions (if applicable)
• Session management
• Rate limiting transparent

Documentation

• Architecture documented
• API documentation complete
• Runbooks created
• Troubleshooting guide
• Configuration documented
• Known limitations listed
• Monitoring runbook
• Incident response plan

Cost Management

• Cost per query calculated
• Budget alerts configured
• Optimization opportunities identified
• Cost attribution working
• Scaling costs understood
• Reserved capacity considered
• Waste identification automated

Operational Excellence

• CI/CD pipeline configured
• Staging environment matches production
• Deployment automation
• Rollback procedures tested
• Change management process
• Incident management plan
• Post-mortem template
• Knowledge base maintained

Compliance & Governance

• Data retention policy defined
• Right to deletion implemented
• Data lineage tracked
• Model versioning
• Prompt versioning
• Audit trail complete
• Regulatory requirements met
• Privacy policy updated

Launch Readiness

• Load testing passed
• Security review completed
• Legal review done
• Stakeholder signoff
• Launch plan documented
• Rollback plan ready
• Support team trained
• Monitoring dashboards live