●Change8

v3.11.0rc1

📦 datadog-sdkView on GitHub →
✨ 13 features🐛 26 fixes⚡ 1 deprecations🔧 28 symbols

Summary

This release introduces significant new features for LLM Observability across various providers (Anthropic, Bedrock, Google GenAI, OpenAI) and adds experimental support for OTLP metrics export. Several bug fixes address tracing accuracy, LLM token counting, and integration stability, while the \`freezegun\` integration is deprecated.

Migration Steps

  1. If you rely on CI Visibility code coverage collection with \`pytest\`, you can now remove \`coverage.py\` as a dependency.
  2. If you were using the \`freezegun\` integration, note that it is deprecated and no longer required for correct test duration reporting; remove reliance on it.
  3. To enable experimental OTLP metrics export, set the environment variable \`DD_METRICS_OTEL_ENABLED\` to \`true\` and ensure your application includes an OTLP metrics exporter.
  4. To disable automatic trace attribute injection into standard logs, set the environment variable \`DD_LOGS_INJECTION\` to \`False\`.

✨ New Features

  • LLM Observability: Added support to \`submit_evaluation_for()\` for submitting boolean metrics in LLMObs evaluation metrics, using \`metric_type="boolean"\`.
  • LLM Observability: Bedrock Converse \`toolResult\` content blocks are formatted as tool messages on LLM Observability \`llm\` spans' inputs.
  • LLM Observability: Introduces capturing the number of input tokens read and written to the cache for Anthropic prompt caching use cases.
  • LLM Observability: Introduces the ability to track the number of tokens read and written to the cache for Bedrock Converse prompt caching.
  • LLM Observability: Adds support to automatically submit Google GenAI calls to LLM Observability.
  • LLM Observability: Introduces tracking cached input token counts for OpenAI chats/responses prompt caching.
  • LLM Observability: Adds support to automatically submit PydanticAI request spans to LLM Observability.
  • MCP: Adds tracing support for \`mcp.client.session.ClientSession.call_tool\` and \`mcp.server.fastmcp.tools.tool_manager.ToolManager.call_tool\` methods in the MCP SDK.
  • OTEL: Adds experimental support for exporting OTLP metrics via the OpenTelemetry Metrics API (requires setting \`DD_METRICS_OTEL_ENABLED\` to \`true\` and including an OTLP metrics exporter).
  • ASGI: Obfuscate resource names on 404 spans when \`DD_ASGI_OBFUSCATE_404_RESOURCE\` is enabled.
  • Code Origin: added support for in-product enablement.
  • Logging: Automatic injection of trace attributes into logs is now enabled for the standard logging library when using either \`ddtrace-run\` or \`import ddtrace.auto\` (can be disabled via \`DD_LOGS_INJECTION=False\`).
  • Google GenAI: Adds support for APM/LLM Observability tracing for Google GenAI's \`embed_content\` methods.

🐛 Bug Fixes

  • CI Visibility: Resolved an issue where \`freezegun\` would not work with tests defined in \`unittest\` classes.
  • CI Visibility: Fixed issue where Test Optimization together with external retry plugins (like \`flaky\` or \`pytest-rerunfailures\`) caused incorrect test result reporting (Test Optimization advanced features disabled when using such plugins).
  • CI Visibility: Resolved an issue where test retry numbers were not reported correctly when tests were run with \`pytest-xdist\`.
  • AAP: Fixed FastAPI body extraction in async contexts for large bodies by setting request body chunk read timeout to 0.1 seconds (configurable via \`DD_FASTAPI_ASYNC_BODY_TIMEOUT_SECONDS\`).
  • Lib Injection: Fix a bug preventing the Single Step Instrumentation (SSI) telemetry forwarder from completing when debug logging was enabled.
  • LiteLLM: Fixed tagging of potentially sensitive parameters as metadata on LLM Observability spans by using an allowlist instead of a denylist.
  • LLM Observability: Addressed upstream issue in Anthropic prompt caching where input tokens were reported incorrectly; now correctly counts input tokens including cached read/write prompt tokens.
  • LLM Observability (OpenAI): Resolved \`AttributeError\` while parsing \`NoneType\` streamed chunk deltas during OpenAI tracing.
  • LLM Observability (OpenAI): Fixed \`AttributeError\` when parsing token metrics for streamed reasoning responses from the responses api.
  • LLM Observability: Addressed upstream issue in Bedrock prompt caching where input tokens were reported incorrectly; now correctly counts input tokens including cached read/write prompt tokens.
  • LLM Observability: Fixed issue where input messages for tool messages were not being captured properly.
  • LLM Observability: Resolved index error with LLM Observability tracing when incomplete streamed responses were returned from OpenAI responses API.
  • LLM Observability: Fixed broken LangGraph span links for execution flows for \`langgraph>=0.3.22\`.
  • LLM Observability: Resolved issue where tool choice input messages for OpenAI Chat Completions were not being captured.
  • LLM Observability: Fixed \`ValueError\` raised when grabbing token values for some providers through \`langchain\` libraries.
  • LLM Observability: Resolved issue where passing back tool call results to OpenAI Chat Completions caused an error with LLM Observability tracing enabled.
  • Dynamic Instrumentation: Improved support for function probes with frameworks interacting with the Python garbage collector (e.g. synapse).
  • Logging: Fixed issue where \`dd.*\` properties were not injected onto logging records unless \`DD_LOGS_ENABLED=true\` was set, affecting non-structured loggers.
  • Profiling: Fixed an issue with greenlet support that could cause greenlet spawning to fail in rare cases.
  • Profiling: Fixed a bug where profile frames from the package specified by DD_MAIN_PACKAGE were marked as "library" code in the profiler UI.
  • Tracing: Resolved issue where programmatically set span service names would not get reported to Remote Configuration.
  • Tracing: Fixed issue where the \`@tracer.wrap()\` decorator failed to preserve the decorated function's return type, returning \`Any\` instead of the original return type.
  • Tracing: Resolved issue where spans had incorrect timestamps and durations when \`freezegun\` was in use.
  • Tracing: Fixed issue in which span durations or start timestamps exceeding the platform's \`LONG_MAX\` caused traces to fail to send.
  • Code Security (IAST): Improved compatibility with \`eval()\` when used with custom globals and locals by ensuring dynamically defined functions are accessible.
  • Code Security (IAST): Addressed AST analysis failures/unexpected behavior when code overrides Python built-ins or globals at runtime (e.g., \`mysqlsh\`).

🔧 Affected Symbols

submit_evaluation_formcp.client.session.ClientSession.call_toolmcp.server.fastmcp.tools.tool_manager.ToolManager.call_toolddtrace-runddtrace.autofreezegunpytestcoverage.pyflakypytest-rerunfailurespytest-xdistFastAPIDD_FASTAPI_ASYNC_BODY_TIMEOUT_SECONDSSingle Step Instrumentation (SSI)litellmAnthropic prompt cachingBedrock ConverseGoogle GenAI embed_contentPydanticAIopenaiLangGraphlangchaineval()DD_METRICS_OTEL_ENABLEDDD_ASGI_OBFUSCATE_404_RESOURCEDD_LOGS_INJECTIONDD_MAIN_PACKAGE@tracer.wrap()

⚡ Deprecations

  • CI Visibility: The \`freezegun\` integration is deprecated and will be removed in 4.0.0. The \`freezegun\` integration is not necessary anymore for the correct reporting of test durations and timestamps.