litellm
Python SDK, Proxy Server (AI Gateway) to call 100+ LLM APIs in OpenAI (or native) format, with cost tracking, guardrails, loadbalancing and logging. [Bedrock, Azure, OpenAI, VertexAI, Cohere, Anthropi
Star Growth
Overview
LiteLLM是一个强大的Python SDK和AI网关代理服务器，为开发者提供统一的接口来调用100多个大型语言模型API。它最大的价值在于标准化了不同AI提供商的调用方式，让开发者可以使用OpenAI格式的代码调用Anthropic Claude、Google VertexAI、AWS Bedrock、Azure OpenAI等各种模型。该工具不仅提供模型调用功能，还集成了成本追踪、安全防护栏、负载均衡和详细日志记录等企业级特性。LiteLLM支持聊天补全、嵌入向量、图像生成、音频处理等多种AI功能端点，是构建多模型AI应用的理想选择。拥有超过4万GitHub星标，属于Y Combinator W23批次公司，在AI基础设施领域具有较高声誉和活跃的开发社区。
Deep Analysis
Unlike OpenRouter (hosted-only routing), LiteLLM is self-hostable and provides a full gateway with per-user spend tracking, virtual keys, and A2A/MCP protocol support — making it the enterprise LLM traffic controller
⚡ Capabilities
- • Unified OpenAI-format API for 100+ LLM providers (Bedrock, Azure, Anthropic, Groq, etc.)
- • AI Gateway proxy server with authentication, rate limiting, and spend tracking per project/user
- • Router with retry/fallback logic across multiple deployments
- • A2A (Agent-to-Agent) protocol support for LangGraph, Vertex AI, Bedrock agents
- • MCP (Model Context Protocol) gateway bridging MCP servers to any LLM
- • 8ms P95 latency at 1000 RPS with load balancing
🔗 Integrations
✓ Best For
- ✓ ML platform teams managing multi-provider LLM access with centralized cost tracking and auth
- ✓ Developers switching between LLM providers without changing application code
✗ Not Ideal For
- ✗ Simple single-provider apps — direct SDK is simpler and faster
- ✗ Self-contained agent building — use LangGraph or CrewAI instead
Languages
Deployment
Pricing Detail
⚠ Known Limitations
- ⚠ Python-only SDK — no native JavaScript/Go client
- ⚠ Proxy server adds a network hop and potential latency for simple direct-call use cases
- ⚠ Enterprise features (SSO, advanced analytics) require paid tier
Pros
- + 统一API接口设计，一套代码兼容100多个不同的LLM提供商，大幅简化多模型切换和对比测试
- + 内置企业级功能如成本追踪、负载均衡、安全防护栏，为生产环境提供完整的AI治理解决方案
- + 既提供Python SDK又提供独立的代理服务器部署模式，适合不同规模和架构的项目需求
Cons
- - 作为中间层抽象，可能无法完全利用某些模型提供商的独特功能和高级参数配置
- - 依赖网络连接和第三方API稳定性，增加了系统的复杂度和潜在故障点
- - 对于简单的单模型应用场景可能存在过度设计，增加不必要的依赖和学习成本
Use Cases
- • AI应用开发中需要对比测试多个LLM模型性能，快速切换不同提供商而无需重写代码
- • 企业级AI服务需要统一的成本监控、访问控制和负载均衡管理多个模型调用
- • 构建AI代理或聊天机器人时需要根据用户需求和成本考虑动态选择最适合的模型