Overview
Xinference (Xorbits Inference) is a powerful platform designed to serve language, speech recognition, and multimodal models. It enables running inference using any open-source LLMs, embedding models, and multimodal models either in the cloud or on-premises.
Key Features for Vector Embeddings
- Embedding Support: Creates text embeddings for vector search applications
- LangChain Integration: XinferenceEmbeddings class for seamless integration
- Multiple Models: Supports various embedding models for different use cases
Inference Capabilities
- OpenAI-Compatible API: RESTful API including Function Calling
- Multiple Interfaces: RPC, CLI, and WebUI for model management
- Distributed Deployment: Seamless distribution across multiple devices or machines
- Automatic Batching: Concurrent requests automatically batched for higher throughput
Architecture
XorbitsAI builds production-ready infrastructure for:
- Model Inference at scale
- AI Agent orchestration
- Intelligent system deployment
- Cloud deployment
- On-premises installation
- Edge computing scenarios
Use Cases
- Large-scale embedding generation
- Multi-model AI applications
- Enterprise LLM deployment
- Vector search pipelines
- Distributed AI inference
