High-throughput, low-latency serving engine for text embeddings, reranking models, CLIP, CLAP and ColPali with GPU acceleration support for local deployment and production use.
Зареждане на още......
На тази страница
Infinity
Infinity is a high-throughput, low-latency REST API serving engine designed for deploying text-embeddings, reranking models, CLIP, CLAP, and ColPali models into production environments.
--device-id 0,1,2,3 for approximately 4x throughput increasedocker run -it --gpus all \
-v $volume:/app/.cache \
-p $port:$port \
michaelf34/infinity:latest \
v2 \
--model-id $model \
--port $port
michaelf34/infinity:latestmichaelf34/infinity:latest-cpupip install infinity-emb
infinity_emb v2 --model-id <model> --port <port>
Use AsyncEmbeddingEngine for programmatic access with maximum flexibility
{url}:{port}/docs for testingFree and open-source, available on GitHub.
Разгледайте още елементи, свързани с този