●Change8

v4.1.0rc2

Breaking Changes
📦 datadog-sdk
⚠ 2 breaking✨ 27 features🐛 18 fixes⚡ 3 deprecations🔧 12 symbols

Summary

This release introduces significant new features in Profiling and LLM Observability, including support for new Python primitives and multi-run experiments. It also removes the aioredis integration and drops support for 32-bit Linux environments.

⚠️ Breaking Changes

  • 32-bit linux is no longer supported. Users must upgrade to a 64-bit environment.
  • The aioredis integration has been removed. Users relying on this integration must find an alternative or remove the dependency.

Migration Steps

  1. If you are using 32-bit linux, you must migrate to a 64-bit environment.
  2. If you were using the removed aioredis integration, replace it with an alternative solution.
  3. If you were using programmatic tracing configuration for Tornado via `ddtrace.contrib.tornado`, switch to configuring tracing using environment variables and `import ddtrace.auto`.
  4. For multi-run experiments, update code to use `ExperimentResult.runs` instead of accessing `rows` or `summary_evaluations` directly.
  5. When starting Ray clusters, use `ddtrace-run` to ensure full instrumentation: `DD_PATCH_MODULES="ray:true,aiohttp:false,grpc:false,requests:false" ddtrace-run ray start --head`.

✨ New Features

  • LLM Observability: Experiments spans now contain metadata from the dataset record.
  • LLM Observability: Experiments spans' input, output, expected_output fields are now emitted as is, allowing object data to be searchable in Datadog.
  • LLM Observability: Experiments spans and children spans are now tagged with human readable names (`dataset_name`, `project_name`, `project_id`, `experiment_name`) for better analysis.
  • Profiling: Added support for `threading.BoundedSemaphore` locking type profiling.
  • Profiling: Added support for `threading.Semaphore` locking type profiling, correctly marking internal locks.
  • Profiling: Added support for Python 3.14 in the Continuous Profiler.
  • Profiling: Added the `process_id` tag to profiles.
  • Profiling: The stack sampler supports async generators and `asyncio.wait`.
  • Profiling: Shows fully qualified name of functions using `codeobject.co_qualname` in memory profiler and lock profiler flamegraphs for Python 3.11+.
  • Profiling: Introduces tracking for the `asyncio.as_completed` util in the Profiler.
  • Profiling: Introduces tracking for `asyncio.wait` in the Profiler.
  • AAP: Attach Application and API Protection findings on API Gateway inferred spans to enable AppSec API Catalog coverage of lambda functions.
  • AAP: Introduces proper support for API10 for redirected requests on urllib3.
  • Anthropic: Adds support for the Anthropic Beta client API (`client.beta.messages.create()` and `client.beta.messages.stream()`, requires Anthropic client version 0.37.0+).
  • Aiokafka: Adds DSM instrumentation support and instrumentation support for `aioredis>=0.9.0`.
  • Added support for uWSGI with gevent when threads are also patched; `thread=False` is no longer required with `gevent.monkey.patch_all`.
  • LLM Observability: Reasoning token counts are now captured from Google GenAI responses.
  • LLM Observability: OpenAI integration captures prompt metadata (id, version, variables, chat template) for reusable prompts using the `responses` endpoint (OpenAI SDK >= 1.87.0).
  • LLM Observability: Experiments can now be run multiple times using the optional `runs` argument, accessible via `ExperimentResult.runs`.
  • LLM Observability: Non-root experiment spans are now tagged with experiment ID, run ID, and run iteration tags.
  • LLM Observability: Adds additional tags to MCP client session and tool call spans.
  • LLM Observability: Reasoning token counts are now captured from OpenAI and OpenAI Agents responses.
  • OpenAI: Introduces support for capturing server-side MCP tool calls invoked via the OpenAI Responses API as a separate span.
  • Langchain: Adds support for tracing `RunnableLambda` instances.
  • MCP: Marks client mcp tool call spans as errors when the corresponding server tool call errored.
  • Crashtracker: Introduces a fallback to capture runtime stack frames when Python's `_Py_DumpTracebackThreads` function is not available.
  • ASGI: Enable context propagation between websocket message spans.

🐛 Bug Fixes

  • Avro: Fixes an issue where Avro instrumentation does not return method results when DSM is enabled.
  • Crashtracker: Fixes missing env variables inheritance for receiver process.
  • Dynamic instrumentation: Fix issue with line probes matching the wrong source file when multiple source files from different Python path entries share the same name.
  • Dynamic instrumentation: Uploading snapshots now retries on all HTTP error codes.
  • Exception replay: Fixed the order in which frames are captured to ensure values close to the initial exception are attached to relevant spans.
  • Exception replay: Fixed an infinite loop causing memory leaks during exception capture, improving speed and memory performance.
  • Exception replay: Ensure exception information is captured when exceptions are raised by the GraphQL client library.
  • Code Security: Fixes critical memory safety issue in IAST when used with forked worker processes (segmentation faults due to stale PyObject pointers).
  • OpenAI: Resolves an issue where instantiating an OpenAI client with a non-string API key resulted in parsing issues.
  • Tracing: Fixed a potential `IndexError` in partial flush when the finished span counter was out of sync.
  • Tracing: `DD_TRACE_PARTIAL_FLUSH_MIN_SPANS` values less than 1 now default to 1 with a warning.
  • Tracing: Resolves a potential deadlock when forking.
  • Tracing: CI Visibility: Ensure the http connection is correctly reset in all error scenarios.
  • Ray: Resolves an issue where Ray jobs not explicitly calling `ray.init()` were not properly instrumented.
  • AAP: Fixes an issue where the appsec layer was not compatible with the lambda/serverless version of the tracer.
  • Lib-injection: Do not inject into the `gsutil` tool.
  • LLM Observability: Fixes an issue where `LLMObs.export_span()` would raise when LLMObs is disabled.
  • LLM Observability: Resolves an issue where `self` was being annotated as an input parameter using LLM Observability function decorators.

🔧 Affected Symbols

ddtrace.contrib.tornadoExperimentResultthreading.BoundedSemaphorethreading.Semaphoreasyncio.waitasyncio.as_completedclient.beta.messages.createclient.beta.messages.streamaiokafkagevent.monkey.patch_allLLMObs.export_spancodeobject.co_qualname

⚡ Deprecations

  • Support for Tornado versions older than v6.1 is deprecated. Users should upgrade to Tornado v6.1 or later.
  • Programmatic tracing configuration via the `ddtrace.contrib.tornado` module is deprecated. Users should configure tracing using environment variables and `import ddtrace.auto` instead.
  • The `ExperimentResult` class' `rows` and `summary_evaluations` attributes are deprecated and will be removed in the next major release. Use the `ExperimentResult.runs` attribute instead to access experiment results and summary evaluations for multi-run experiments.