LiteLLM
AI & LLMsPython SDK, Proxy Server (AI Gateway) to call 100+ LLM APIs in OpenAI (or native) format, with cost tracking, guardrails, loadbalancing and logging. [Bedrock, Azure, OpenAI, VertexAI, Cohere, Anthropic, Sagemaker, HuggingFace, VLLM, NVIDIA NIM]
Release History
v1.87.0-rc.221 fixes10 featuresThis release focuses heavily on stability, bug fixes across various providers (Bedrock, Cohere, Deepseek, Vertex AI), and significant feature additions including Gemini 3.5 Flash support and enhanced Prometheus metrics. It also introduces instructions for verifying Docker image signatures.
v1.86.11 fixThis patch release introduces cryptographic signing for all LiteLLM Docker images using cosign and includes a fix for a non-root npm package issue.
v1.86.011 fixes13 featuresThis release introduces significant infrastructure updates, including componentization of gateway/UI services and new Weighted-Routing Failover capabilities. It also addresses several bugs across providers and enhances OpenTelemetry tracing support.
v1.87.0-rc.122 fixes14 featuresThis release introduces Docker image signing via cosign for enhanced security and adds numerous fixes across providers like Gemini, Vertex AI, and Bedrock. Key features include new model cost maps, UI improvements, and support for the Microsoft Purview DLP guardrail.
v1.85.11 featureThis release introduces cryptographic signing for all LiteLLM Docker images using cosign to enhance supply chain security and provides instructions for signature verification.
v1.84.11 featureThis release introduces cryptographic signing for all LiteLLM Docker images using cosign to improve supply chain security. Users are provided with instructions on how to verify these signatures.
v1.87.0-dev.16 fixes5 featuresThis release introduces Docker image signature verification via cosign and adds support for Gemini 3.5 Flash and Gemini managed agents. It also includes numerous fixes across Bedrock, Deepseek, and proxy streaming.
v1.86.0-rc.112 fixes13 featuresThis release introduces Docker image signing via cosign for enhanced security and adds significant features like Weighted-Routing Failover and componentized service architecture. Numerous bug fixes address validation, rate limiting, and provider integrations across Vertex AI, Bedrock, and MCP.
v1.85.032 fixes5 featuresThis release focuses heavily on security hardening, including fixing multiple SSRF vulnerabilities and tightening access controls across various components. New features include support for the Z.AI GLM-5 model and enhancements to Gemini multimodal embeddings.
v1.84.0Breaking23 fixes11 featuresThis release introduces new features like support for gpt-image-2 and AIHubMix provider, alongside numerous bug fixes across caching, logging, and provider integrations. It also contains a breaking change: the removal of the /ui/chat page.
v1.85.0-rc.232 fixes5 featuresThis release focuses heavily on security hardening, fixing multiple SSRF vulnerabilities and tightening access controls across proxy, guardrails, and various clients. It also introduces new model support for Z.AI on Bedrock and enhances Gemini multimodal embedding capabilities.
v1.83.14-stable.patch.31 featureThis release introduces cryptographic signing for all LiteLLM Docker images using cosign to improve supply chain security. Users are provided with instructions on how to verify image signatures.
v1.83.10-stable.patch.11 featureThis release introduces cryptographic signing for all LiteLLM Docker images using cosign to improve supply chain security. Instructions are provided for verifying these signatures.
v1.83.14-stable.patch.22 featuresThis release introduces cryptographic signing for all LiteLLM Docker images using cosign to improve supply chain security. Instructions are provided for verifying these signatures.
v1.84.0-rc.126 fixes11 featuresThis release introduces Docker image signature verification via cosign and adds several new features, including support for gpt-image-2 and AIHubMix provider. Numerous bug fixes address issues related to caching, logging, Vertex AI, and proxy stability.
v1.83.14-stable.patch.11 featureThis release introduces cryptographic signing for all LiteLLM Docker images using cosign to improve supply chain security. Instructions are provided for verifying image authenticity.
v1.83.14-stable25 fixes9 featuresThis release focuses heavily on security by introducing Docker image signing verification via cosign and includes numerous fixes across various providers (Anthropic, Azure, Gemini, Bedrock) and proxy features, alongside new model support like GPT-5.5.
1.84.0-dev.220 fixes9 featuresThis release introduces Docker image signing verification via cosign and adds support for new models like gpt-image-2 and providers like AIHubMix. Numerous security hardening measures and bug fixes were applied across proxy, authentication, and vector store integrations.
1.84.0-dev.113 fixes4 featuresThis release introduces Docker image signing verification via cosign and includes numerous fixes across caching, logging, pricing, and provider integrations. A feature to lazy-load optional routers was introduced and subsequently reverted.
v1.83.14.rc.127 fixes8 featuresThis release introduces Docker image signing via cosign for enhanced security and adds support for new models like GPT-5.5/5.4 snapshots and Bedrock GLM-5. Numerous fixes address streaming issues, model pricing accuracy, and security hardening across proxy and authentication layers.
v1.83.10-stable1 featureThis release introduces cryptographic signing for all LiteLLM Docker images using cosign to improve supply chain security. Instructions are provided for verifying image authenticity.
v1.83.13-nightly14 fixes5 featuresThis release introduces Docker image signature verification via cosign and adds features like Dashscope image generation support and multi-region Vertex hosts. Numerous bug fixes address issues across providers, UI filtering, and budget tracking.
v1.83.7-stable.patch.11 featureThis release introduces cryptographic signing for all LiteLLM Docker images using cosign to allow users to verify image authenticity. Instructions for verification using commit hash or release tag are provided.
v1.83.12-nightly6 fixes2 featuresThis release introduces Docker image signing via Cosign for enhanced security and adds support for the moonshot/kimi-k2.6 model. It also includes several infrastructure updates and bug fixes related to spend tracking and Docker builds.
v1.83.11-nightly8 fixes6 featuresThis release introduces Docker image signing verification via cosign and adds several features including proxy hot-reloading and audio support for Scaleway. It also includes numerous infrastructure improvements and bug fixes across Bedrock integration and CI stability.
v1.83.7-stable21 fixes9 featuresThis release introduces enhanced Docker image security via cosign signature verification and adds several new features, including AWS Gov Cloud support and new proxy management endpoints. Numerous bug fixes address issues related to database cleanup, routing logic, and Dockerfile consistency.
v1.83.10-nightlyBreaking17 fixes8 featuresThis release introduces mandatory Python 3.10+ support and enhances security by implementing Docker image signing verification via Cosign. New features include support for grok-4.20-0309-reasoning and improved budget management capabilities.
v1.83.9-nightly22 fixes4 featuresThis release introduces Docker image signature verification via cosign and includes numerous bug fixes across proxy security, caching, Bedrock handling, and UI components. Infrastructure updates also stabilized CI/CD processes.
v1.83.8-nightly20 fixes10 featuresThis release introduces Docker image signing via cosign for verification and adds new features like BM25-based prompt compression (litellm.compress()) and advisor tool orchestration. Numerous bug fixes address issues across S3 signing, caching, UI redirects, and provider integrations like Vertex AI and Dashscope.
v1.83.3-stable29 fixes5 featuresThis release focuses heavily on bug fixes across various providers (Anthropic, Gemini, Vertex AI) and internal proxy/UI stability, alongside introducing Docker image signing verification via cosign.
v1.83.7.rc.1Breaking14 fixes7 featuresThis release introduces Docker image signing verification instructions and includes numerous bug fixes across proxy, S3, logging, and guardrails. It also features a breaking change reducing the default Prometheus latency histogram buckets.
v1.83.6-nightly13 fixes7 featuresThis release introduces Docker image signature verification using cosign, adds new API endpoints and features like Ramp callbacks, and includes numerous bug fixes across routing, cost calculation, and Docker builds.
v1.82.3-stable.patch.41 featureThis release introduces cryptographic signing for all LiteLLM Docker images using cosign to improve supply chain security. Users are encouraged to verify image authenticity using the provided signing instructions.
v1.82.3.dev.91 featureThis release introduces cryptographic signing for all LiteLLM Docker images using cosign to improve supply chain security. Instructions are provided for verifying image signatures.
v1.83.3.rc.111 fixes6 featuresThis release introduces Docker image signature verification via cosign and adds significant features for team management, including project-level guardrails and enhanced UI controls for keys and rate limits. Numerous bug fixes address issues across the UI, proxy, and agent integrations.
v1.82.3-stable.patch.36 fixes3 featuresThis patch release introduces new UI features for rate limiting and credential exposure, adds embedding usage tracking fallback, and includes several UI and Docker build bug fixes. It also details how to verify the integrity of LiteLLM Docker images using cosign.
v1.82.6.rc.49 fixes4 featuresThis release introduces Docker image signing verification using cosign and adds significant new features to the router, including order-based fallback and health-check-driven routing. Several routing and Docker build fixes were also implemented.
v1.83.2-nightly1 featureThis release introduces Docker image signing using cosign for improved security verification. It also includes minor version bumps for internal components.
v1.83.5-nightly17 fixes9 featuresThis release introduces Docker image signing verification via cosign and adds significant features to the UI and Proxy, including project-level guardrails and Azure Entra ID credential support. Several bug fixes address issues related to UI filtering, team management, and Docker image loading.
v1.82.3.dev.71 featureThis release introduces Docker image signing using cosign to ensure image integrity. Users can now verify the authenticity of LiteLLM Docker images before deployment.
v1.82.3.dev.61 featureThis release introduces Docker image signing using cosign to ensure image integrity. Users can now verify the authenticity of LiteLLM Docker images before deployment.
v1.83.1-nightlyBreakingThis was a test release from the LiteLLM team to test a new signing process. Co-sign verification is temporarily non-functional.
v1.83.0-nightly14 fixes11 featuresThis release focuses heavily on infrastructure hardening, security improvements (like OpenSSF Scorecard adoption and dependency pinning), and numerous bug fixes across providers like Gemini, Anthropic, and Azure OpenAI. New features include Prometheus metrics for batch operations and budget enforcement across multi-pod deployments.
v1.82.3-stable.patch.29 fixes5 featuresThis release focuses heavily on UI enhancements, including new organizational and team management features, alongside critical bug fixes related to client handling, logging redaction, and database migration stability. A new provider for Amazon Nova models on SageMaker was also introduced.
v1.82.6.dev21 fix2 featuresThis release introduces enhancements to Prometheus metrics and proxy callback tracking, alongside a fix for missing post-call guardrail logs in the proxy.
v1.82.6.rc.2This release appears to be a release candidate (v1.82.6.rc.2) with no detailed changes provided in the snippet, linking only to the comparison view.
v1.82.3-stable.patch.18 fixes5 featuresThis release introduces support for the sagemaker_nova provider and includes numerous UI enhancements related to key management and team settings. Several critical bug fixes address issues with client eviction, logging redaction, and database migration stability.
v1.82.3.dev.5This release contains internal development changes between versions v1.82.5.dev.1 and v1.82.3.dev.5.
v1.82.5.dev.133 fixes11 featuresThis release focuses heavily on bug fixes and stability improvements across various providers like Anthropic, Gemini, and Vertex AI, alongside introducing new features such as Akto Guardrails integration and prompt management support for the Responses API.
v1.82.6.dev1This release appears to be a development release (v1.82.6.dev1) following a release candidate (v1.82.6.rc.1), with the full changelog available via the provided comparison link.
v1.82.6.rc.127 fixes10 featuresThis release introduces new features like the sagemaker_nova provider and various UI enhancements, alongside numerous bug fixes across providers (Anthropic, Gemini, Vertex AI) and infrastructure improvements, including security updates and better logging redaction.
v1.82.6.dev.118 fixes8 featuresThis release focuses heavily on UI modernization, especially around Teams and MCP management, alongside numerous bug fixes for proxy guardrails, logging, and key management endpoints. It also introduces infrastructure improvements and new environment variable support for Anthropic.
v1.82.6-nightly34 fixes9 featuresThis release focuses heavily on stability and feature parity across various providers, including significant fixes for Anthropic reasoning summaries, Gemini image handling, and Vertex AI pricing/streaming. It also introduces new features like Akto Guardrails integration and control plane management.
v1.82.1.dev.12 fixesThis development release primarily focuses on fixing bugs related to parameter passing for Azure OpenAI and improving streaming response handling across various providers.
v1.81.14.dev.39 fixes5 featuresThis release focuses heavily on UI modernization, infrastructure improvements including a control plane for worker management, and numerous fixes across various providers like Bedrock, Gemini, and Moonshot. Several documentation updates regarding pricing and caching were also included.
v1.81.14.dev.29 fixes5 featuresThis release focuses heavily on UI modernization, especially around Teams features, and includes numerous fixes across various providers like Bedrock, Gemini, and Anthropic. A new control plane for multi-proxy worker management was also introduced.
v1.82.3.dev.49 fixes5 featuresThis release focuses heavily on UI enhancements, including new organizational and team management features, alongside critical bug fixes related to logging redaction, database migrations, and dependency security updates.
v1.82.5-nightly15 fixes9 featuresThis release focuses heavily on bug fixes across the proxy, UI, and logging systems, alongside infrastructure improvements like black formatting and dependency updates. Key features include enhanced Anthropic integration and new control plane capabilities for multi-proxy management.
v1.82.3.dev.319 fixes7 featuresThis release focuses heavily on UI enhancements, including new organizational and team management features, alongside numerous bug fixes addressing security, logging redaction, and database migration stability. A new provider, sagemaker_nova, was also introduced.
v1.81.14.dev.117 fixes4 featuresThis release focuses heavily on bug fixes across the proxy, UI, and logging systems, alongside infrastructure updates like code formatting and dependency regeneration. New features include enhanced team endpoint access control and improved Anthropic environment variable support.
v1.82.3.dev.221 fixes4 featuresThis release focuses heavily on bug fixes across proxy functionality, UI components, and provider integrations, alongside infrastructure updates like dependency regeneration and code formatting. New features include enhanced team listing access control and proxy-wide rate limits.
v1.82.3.dev.12 fixesThis development release primarily focuses on fixing parameter passing issues for the Azure OpenAI provider, specifically for max_tokens and temperature.
v1.82.4-nightly22 fixes7 featuresThis release focuses heavily on UI enhancements, including new organizational controls and modernized settings pages. It also includes numerous bug fixes addressing security issues, database migration stability, and provider integration reliability.
v1.82.0.dev.7This release is a stable patch for Litellm version 1820, implemented by @shivamrawat1.
v1.82.3-stable10 fixes5 featuresThis release focuses heavily on UI enhancements, including organization and team management features, alongside critical bug fixes related to client handling, logging redaction, and database migration stability. A new provider for Amazon Nova models on SageMaker was also introduced.
v1.82.3.rc.19 fixes5 featuresThis release focuses heavily on UI enhancements, including organization and team management features, alongside critical bug fixes related to logging redaction, database migrations, and dependency security updates.
v1.82.0.patch520 fixes4 featuresThis patch release focuses heavily on bug fixes across various integrations (Anthropic, Azure, HuggingFace) and infrastructure stability, alongside introducing a new User Info V2 Endpoint and allowing organization_id setting on key updates.
v1.82.2-dynamoai.dev28 fixes9 featuresThis release focuses heavily on bug fixes, infrastructure improvements, and significant updates to the UI, including new endpoints and administrative features. Key fixes address security vulnerabilities, proxy stability, and various provider integrations.
v1.82.2-focus-export-2.dev21 fixes12 featuresThis release introduces several new UI features, including user info endpoints and team management tools, alongside numerous bug fixes addressing issues in proxy stability, cost tracking, and API integrations. Dependency updates for security were also performed.
v1.82.2-nightly.dev1This release primarily consists of internal development updates, as indicated by the nightly/dev tag, with a link provided to the detailed comparison.
v1.82.2-silent.dev24 fixes20 featuresThis release focuses heavily on enhancing Multi-Cloud Proxy (MCP) functionality, including new authentication methods, BYOM support, and UI improvements. It also brings numerous bug fixes across providers like Gemini, Anthropic, and OpenAI, alongside new feature support for transcription and image editing.
v1.82.2-nightly27 fixes19 featuresThis release focuses heavily on expanding support for various models (Gemini, Mistral Voxtral, Qwen3.5) and enhancing the Multi-Cloud Proxy (MCP) features, including new authentication and UI workflows. Numerous bug fixes address streaming issues, provider specific mappings, and security concerns.
v1.82.0.dev6This release appears to be a development build (v1.82.0.dev6) referencing the comparison against the previous stable release (v1.82.0-stable), but the specific changes are not detailed in the provided notes.
v1.82.0.dev527 fixes19 featuresThis release introduces significant feature enhancements, particularly around MCP server support, new model integrations (Voxtral, Qwen3.5), and expanded guardrail capabilities. Numerous bug fixes address issues across streaming, Anthropic integration, and UI stability.
v1.82.0.dev4This release primarily points to the full changelog for version v1.82.0.dev4 compared to v1.82.0-stable.
v1.82.0.dev3This release appears to be a development build (v1.82.0.dev3) following v1.82.0-stable, with the full changelog available via the provided comparison link.
v1.82.1-focus-dev14 fixesThis release focuses primarily on bug fixes across various components, including routing, caching, provider integrations (Fireworks, Sagemaker, Vertex AI, Bedrock), and the Responses API streaming bridge. Several provider-specific URL and request body issues were resolved.
v1.82.1-nightly.1-reThis release corresponds to a nightly update identified by the comparison link between v1.82.1-nightly.1 and v1.82.1-nightly.1-re.
v1.82.1.rc.143 fixes10 featuresThis release focuses heavily on bug fixes across numerous providers (Gemini, Anthropic, OpenAI, Azure, etc.), improves streaming performance, and introduces new features like UI Projects and enhanced model cost tracking. A key structural change involves updating response types to use ModelResponseStream instead of StreamingChoices.
v1.82.0-stable1 fixThis stable release primarily addresses a bug where HTTP/SDK clients were incorrectly closed during LLMClientCache eviction.
v1.82.1-nightly.2This release primarily consists of internal updates and bug fixes, referenced by the full changelog link.
v1.82.1-azure-dev43 fixes10 featuresThis release focuses heavily on bug fixes across numerous providers (Gemini, Anthropic, OpenAI, Azure, etc.), improves streaming performance, and introduces new features like UI Projects and enhanced model cost tracking. It also deprecates older xAI models.
v1.82.1-nightly.1This release primarily consists of internal updates and links to the detailed comparison between nightly builds.
v1.82.1-silent-dev214 fixes2 featuresThis release focuses heavily on bug fixes across various integrations, including completion bridges, routing, proxy endpoints, caching, and specific provider integrations like Vertex AI, Fireworks, Sagemaker, and Bedrock. New features include silent metric recording and helmchart deployment strategy support.
v1.82.0.patch42 fixesThis patch release primarily focuses on bug fixes, specifically addressing incorrect parameter passing for Azure OpenAI and improving streaming response handling across providers.
v1.82.1-dev13 fixesThis release focuses heavily on bug fixes across various components including routing, caching, proxy endpoints, and specific provider integrations like Fireworks, Sagemaker, and Vertex AI. Key improvements involve error handling during retries and data preservation in streaming responses.
1.82.1-dev-22 fixesThis development release focuses on fixing bugs related to parameter passing for Azure OpenAI and improving streaming response handling across various providers.
v1.82.1-nightly43 fixes10 featuresThis release focuses heavily on bug fixes across numerous providers (Gemini, Anthropic, OpenAI, Azure, etc.), improves streaming performance, and introduces new features like UI Projects and enhanced model cost tracking. A key change involves updating response types to use ModelResponseStream instead of StreamingChoices.
v1.82.dev43 fixes10 featuresThis release focuses heavily on bug fixes across various providers (Gemini, Anthropic, OpenAI, Azure AI, etc.), enhances model cost mapping, and introduces new features like UI Projects and improved streaming latency. It also deprecates older xAI models.
v1.82.rc.32 fixesThis release focuses on fixing critical parameter passing issues for the Azure OpenAI provider, specifically for max_tokens and timeout settings.
v1.81.14-stable.gpt-5.4-patch7This release contains a patch update for the v1.81.14-stable.gpt-5.4 branch, with details available in the full changelog link.
v1.82.rc.2This release corresponds to release candidate 2 (v1.82.rc.2) following the v1.82.0-nightly build.
v1.82.0.patch32 fixesThis patch release primarily focuses on fixing an issue related to Azure OpenAI API key passing and resolving a streaming max_tokens bug.
v1.81.14-stable.gpt-5.4-patch62 fixesThis patch release focuses on fixing critical bugs related to tool call handling for the Gemini 1.5 Flash model on the completion endpoint.
v1.82.dev243 fixes10 featuresThis release focuses heavily on bug fixes across numerous providers (Gemini, Anthropic, OpenAI, Azure, etc.), enhances model tracking and cost mapping, and introduces performance improvements for streaming latency. It also adds new UI features for project management and access control.
v1.82.dev141 fixes11 featuresThis release focuses heavily on bug fixes across various providers (Gemini, Anthropic, OpenAI, Azure, OpenRouter) and internal systems like MCP and caching. It also introduces new model support, performance optimizations for spendlogs and streaming latency, and new UI features for project management.
v1.81.14-stable.gpt-5.4-patch5This release corresponds to a patch update in the v1.81.14-stable.gpt-5.4 series, detailed in the linked comparison.
v1.81.14-stable.gpt-5.4_patch21 fixThis patch release primarily addresses a bug related to the incorrect passing of the `max_tokens` parameter to the OpenAI API for specific models.
1.82.143 fixes10 featuresThis release focuses heavily on bug fixes across numerous providers (Gemini, Anthropic, OpenAI, Azure, etc.), improves streaming performance, and introduces new features like UI Projects and enhanced model cost tracking.
v1.81.14-stable.gpt-5.4_patch2 fixesThis patch release primarily addresses bugs related to the Azure OpenAI provider, ensuring correct handling of completion and streaming responses.
Common Errors
ServiceUnavailableError7 reportsServiceUnavailableError usually indicates the LLM provider is overloaded or temporarily unavailable. Implement retry logic with exponential backoff using `retry` decorator in `litellm` around the failing function calls, and/or check the LLM provider's status page to confirm an outage before retrying. You might also need to increase your rate limits or switch to a different model if the issue persists.
ContentPolicyViolationError3 reportsContentPolicyViolationError usually arises from AI models flagging input or output text as violating their content policies. To fix it, carefully review your prompt and the model's response for potentially harmful or sensitive content. Redraft prompts to be less suggestive or controversial and implement output filtering to sanitize model responses, ensuring compliance with content safety guidelines.
ModuleNotFoundError3 reports"ModuleNotFoundError" usually arises when a required Python package isn't installed, or the Python environment is misconfigured. Resolve this by first ensuring the necessary package is installed using `pip install <package_name>` (e.g., `pip install fastapi`). If the package is installed yet still causing problems, verify your Python environment and that the package version is compatible.
MidStreamFallbackError2 reportsThe MidStreamFallbackError in litellm often arises when a streaming response from an initial model fails (e.g., due to content policy or network issues) and the fallback mechanism within `content_policy_fallbacks` or similar configurations encounters a problem during the stream continuation. To fix this, ensure your fallback models are also capable of streaming and that their configurations are valid; additionally, robust error handling should be implemented within the `streaming_handler` to gracefully manage exceptions during fallback streaming, preventing premature stream termination.
APIConnectionError2 reportsAPIConnectionError usually arises from network issues or temporary unavailability of the hosted LLM endpoint. Fix this by implementing retry mechanisms with exponential backoff and failover logic to redundant deployments within litellm, ensuring robust error handling for temporary API outages. Specifically, add `num_retries` kwargs to embedding model groups to trigger automatic retries and properly handle `APIConnectionError` in cooldown handlers to allow failover to healthy deployments.
BudgetExceededError2 reportsBudgetExceededError in litellm often arises from inaccurate spend tracking, especially with Redis, leading to premature budget exhaustion. To fix this, ensure your Redis instance is correctly configured for persistent data storage and implement idempotent spend counter updates to prevent double-counting across pods or restarts; regularly monitor your spending and Redis counter values to detect discrepancies early.
Related AI & LLMs Packages
AutoGPT is the vision of accessible AI for everyone, to use and to build on. Our mission is to provide the tools, so that you can focus on what matters.
Get up and running with OpenAI gpt-oss, DeepSeek-R1, Gemma 3 and other models.
🦜🔗 The platform for reliable agents.
The most powerful and modular diffusion model GUI, api and backend with a graph/nodes interface.
LLM inference in C/C++
GPT4All: Run Local LLMs on Any Device. Open-source and available for commercial use.
Subscribe to Updates
Get notified when new versions are released