●Change8

v4.1.0rc1

Breaking Changes
📦 datadog-sdkView on GitHub →
⚠ 1 breaking✨ 28 features🐛 13 fixes⚡ 3 deprecations🔧 20 symbols

Summary

This release introduces experimental OpenFeature provider support and significant enhancements to LLM Observability, including richer span tagging and multi-run experiment tracking. It also drops support for 32-bit Linux and removes the deprecated aioredis integration.

⚠️ Breaking Changes

  • 32-bit linux is no longer supported. Users must upgrade to a 64-bit environment to continue using dd-trace-py.

Migration Steps

  1. Upgrade to a 64-bit Linux environment if currently using 32-bit Linux.
  2. If using Tornado, upgrade to Tornado v6.1 or later.
  3. If programmatically configuring Tornado tracing, migrate configuration to use environment variables and `import ddtrace.auto`.
  4. For LLM Observability, update code relying on `ExperimentResult.rows` or `ExperimentResult.summary_evaluations` to use `ExperimentResult.runs` instead.
  5. When using Ray, ensure jobs are started with `ddtrace-run` (e.g., `DD_PATCH_MODULES="ray:true,aiohttp:false,grpc:false,requests:false" ddtrace-run ray start --head`) for complete tracing.

✨ New Features

  • Added OpenFeature provider implementation for feature flag evaluation through the OpenFeature API (experimental).
  • LLM Observability: Experiments spans now contain metadata from the dataset record.
  • LLM Observability: Experiments spans' input, output, expected_output fields are now emitted as is, allowing object data to be searchable in Datadog.
  • LLM Observability: Experiments spans and children spans are now tagged with human readable names: `dataset_name`, `project_name`, `project_id`, `experiment_name`.
  • Profiling: Added support for `threading.BoundedSemaphore` locking type profiling.
  • Profiling: Added support for `threading.Semaphore` locking type profiling, correctly handling internal lock detection.
  • Profiling: Added support for Python 3.14 in the Continuous Profiler.
  • Profiling: Added the `process_id` tag to profiles.
  • Profiling: The stack sampler supports async generators and `asyncio.wait`.
  • Profiling: Shows fully qualified name of functions using `codeobject.co_qualname` in memory profiler and lock profiler flamegraphs for Python 3.11+.
  • Profiling: Introduced tracking for the `asyncio.as_completed` util in the Profiler.
  • Profiling: Introduced tracking for `asyncio.wait` in the Profiler.
  • AAP: Attach Application and API Protection findings on API Gateway inferred spans.
  • AAP: Introduced proper support for API10 for redirected requests on urllib3.
  • Anthropic: Added support for the Anthropic Beta client API (`client.beta.messages.create()` and `client.beta.messages.stream()`, requires Anthropic client >= 0.37.0).
  • Aiokafka: Added DSM instrumentation support and instrumentation support for `aiokafka>=0.9.0`.
  • Added support for uWSGI with gevent when threads are also patched, removing the need for `thread=False` in `gevent.monkey.patch_all`.
  • LLM Observability: Reasoning token counts are now captured from Google GenAI responses.
  • LLM Observability: OpenAI integration captures prompt metadata (id, version, variables, chat template) for reusable prompts via the `responses` endpoint (OpenAI SDK >= 1.87.0).
  • LLM Observability: Experiments can now be run multiple times using the optional `runs` argument, accessible via `ExperimentResult.runs`.
  • LLM Observability: Non-root experiment spans are now tagged with experiment ID, run ID, and run iteration tags.
  • LLM Observability: Added additional tags to MCP client session and tool call spans.
  • LLM Observability: Reasoning token counts are now captured from OpenAI and OpenAI Agents responses.
  • LLM Observability (openai): Introduced support for capturing server-side MCP tool calls invoked via the OpenAI Responses API as a separate span.
  • LangChain: Added support for tracing `RunnableLambda` instances.
  • MCP: Marks client mcp tool call spans as errors when the corresponding server tool call errored.
  • Crashtracker: Introduced a fallback to capture runtime stack frames when Python's `_Py_DumpTracebackThreads` function is unavailable.
  • ASGI: Enabled context propagation between websocket message spans.

🐛 Bug Fixes

  • Avro instrumentation now correctly returns method results when DSM is enabled.
  • Crashtracker now correctly inherits environment variables for the receiver process.
  • Dynamic instrumentation line probes now correctly match source files when multiple files share the same name across different Python path entries.
  • Dynamic instrumentation snapshot uploads now retry on all HTTP error codes.
  • Exception replay fixed the order of frame capture to ensure values close to the exception point are attached to relevant spans.
  • Exception replay fixed an infinite loop causing memory leaks during exception capture, improving speed and memory performance.
  • Exception replay ensures exception information is captured when exceptions are raised by the GraphQL client library.
  • Code Security fixed a critical memory safety issue in IAST when used with forked worker processes (segmentation faults due to stale PyObject pointers).
  • OpenAI: Resolved an issue causing parsing issues when instantiating an OpenAI client with a non-string API key.
  • Tracing fixed a potential `IndexError` in partial flush when span counters were out of sync.
  • Tracing issues a warning and defaults `DD_TRACE_PARTIAL_FLUSH_MIN_SPANS` to 1 if a value less than 1 is provided.
  • CI Visibility: Ensures the http connection is correctly reset in all error scenarios.
  • Ray: Fixed instrumentation failure for Ray jobs that did not explicitly call `ray.init()` by recommending the use of `ddtrace-run`.

🔧 Affected Symbols

ddtrace.contrib.openfeatureExperimentResultddtrace.contrib.tornadothreading.BoundedSemaphorethreading.Semaphorethreading.Conditionasyncio.waitasyncio.as_completedcodeobject.co_qualnameurllib3client.beta.messages.createclient.beta.messages.streamaiokafkagevent.monkey.patch_allclient.messages.createclient.messages.streamRunnableLambdamcp tool call spans_Py_DumpTracebackThreadswebsocket message spans

⚡ Deprecations

  • Support for Tornado versions older than v6.1 is deprecated. Users should upgrade to Tornado v6.1 or later.
  • Programmatic tracing configuration via the `ddtrace.contrib.tornado` module is deprecated. Configuration should now use environment variables and `import ddtrace.auto` instead.
  • The `ExperimentResult` class' `rows` and `summary_evaluations` attributes are deprecated and will be removed in the next major release. Use the `ExperimentResult.runs` attribute instead to access experiment results and summary evaluations.