●Change8

v4.1.0

📦 datadog-sdkView on GitHub →
✨ 25 features🐛 18 fixes⚡ 2 deprecations🔧 17 symbols

Summary

This release introduces significant new features in Continuous Profiling (support for Python 3.14, new lock profiling types) and LLM Observability (multi-run experiments, enhanced tagging). It also deprecates older Tornado versions and resolves several critical bugs related to exception replay, IAST memory safety in forked processes, and tracing initialization in Ray jobs.

Migration Steps

  1. If using Tornado, ensure you are running Tornado version v6.1 or later.
  2. If using LLM Observability, update code referencing `ExperimentResult.rows` or `ExperimentResult.summary_evaluations` to use `ExperimentResult.runs` instead.
  3. When starting Ray clusters, use `ddtrace-run` to ensure full tracing capabilities, e.g., `DD_PATCH_MODULES="ray:true,aiohttp:false,grpc:false,requests:false" ddtrace-run ray start --head`.

✨ New Features

  • Profiling support added for `threading.BoundedSemaphore` locking type.
  • Profiling support added for `threading.Semaphore` locking type, correctly marking internal locks to prevent double-counting.
  • Support added for Python 3.14 in the Continuous Profiler.
  • The `process_id` tag (current process ID) is now added to profiles.
  • The stack sampler now supports async generators and `asyncio.wait`.
  • Fully qualified function names using `codeobject.co_qualname` are now shown in memory profiler and lock profiler flamegraphs for Python 3.11+.
  • Tracking introduced for the `asyncio.as_completed` utility in the Profiler.
  • Tracking introduced for `asyncio.wait` in the Profiler, enabling tracking of dependencies between awaited Tasks/Coroutines.
  • Application and API Protection findings are now attached on API Gateway inferred spans to enable AppSec API Catalog coverage of lambda functions.
  • Proper support introduced for API10 for redirected requests on urllib3.
  • Support added for the Anthropic Beta client API (`client.beta.messages.create()` and `client.beta.messages.stream()`), requiring Anthropic client version 0.37.0 or higher.
  • DSM instrumentation support added for aiokafka.
  • Instrumentation support added for `aiokafka>=0.9.0`.
  • Support added for uWSGI with gevent when threads are also patched; using `thread=False` is no longer required during monkey-patching with gevent via `gevent.monkey.patch_all`.
  • Reasoning token counts are now captured from Google GenAI responses.
  • The OpenAI integration now captures prompt metadata (id, version, variables, and chat template) for reusable prompts when using the `responses` endpoint (OpenAI SDK >= 1.87.0).
  • Experiments can now be run multiple times using the optional `runs` argument, utilizing the new `ExperimentResult.runs` attribute.
  • Non-root experiment spans are now tagged with experiment ID, run ID, and run iteration tags.
  • Additional tags added to MCP client session and tool call spans to power LLM Observability MCP tool call features.
  • Reasoning token counts are now captured from OpenAI and OpenAI Agents responses.
  • Support introduced for capturing server-side MCP tool calls invoked via the OpenAI Responses API as a separate span.
  • Support added for tracing `langchain.RunnableLambda` instances.
  • Client mcp tool call spans are now marked as errors when the corresponding server tool call errored.
  • Crashtracker now falls back to capture runtime stack frames when Python's `_Py_DumpTracebackThreads` function is unavailable.
  • Context propagation enabled between websocket message spans in ASGI.

🐛 Bug Fixes

  • Fixed an issue where Avro instrumentation did not return method results when DSM was enabled.
  • Fixed missing environment variable inheritance for the Crashtracker receiver process.
  • Uploading snapshots now retries on all HTTP error codes in dynamic instrumentation.
  • Fixed the order of frame capture in exception replay to ensure values close to the initial exception point are attached to relevant spans.
  • Fixed an infinite loop causing memory leaks during exception capture in exception replay, improving speed and memory performance.
  • Ensured exception information is captured when exceptions are raised by the GraphQL client library.
  • Fixed critical memory safety issue in IAST when used with forked worker processes (Gunicorn/Uvicorn MCP servers) by preventing segmentation faults from stale PyObject pointers in native taint maps after fork.
  • Resolved an issue where instantiating an OpenAI client with a non-string API key caused parsing issues.
  • Fixed a potential `IndexError` in partial flush when the finished span counter was out of sync with actual finished spans.
  • Values of `DD_TRACE_PARTIAL_FLUSH_MIN_SPANS` less than 1 now default to 1 with a warning.
  • Resolved a potential deadlock when forking.
  • CI Visibility: Ensured the http connection is correctly reset in all error scenarios.
  • Resolved an issue where Ray jobs not explicitly calling `ray.init()` were not properly instrumented; users should use `ddtrace-run` to start Ray clusters.
  • Fixed an issue where the AppSec layer was incompatible with the lambda/serverless version of the tracer.
  • Injection is now skipped for the `gsutil` tool.
  • Fixed an issue where `LLMObs.export_span()` raised an error when LLMObs was disabled.
  • Resolved an issue where `self` was being annotated as an input parameter using LLM Observability function decorators.
  • Fixed an issue where `LLMObs.annotation_context()` properties (tags, prompt, name) were not applied to subsequent LLM operations within the same context block during sequential operations like Langchain batch calls.

🔧 Affected Symbols

ExperimentResultthreading.BoundedSemaphorethreading.Semaphorethreading.Conditionthreading.Lockasyncio.waitasyncio.as_completedurllib3client.beta.messages.createclient.beta.messages.streamaiokafkagevent.monkey.patch_allLLMObs.export_spanLLMObs.annotation_contextRunnableLambda_Py_DumpTracebackThreadsgsutil

⚡ Deprecations

  • Support for Tornado versions older than v6.1 is deprecated. Users must use Tornado v6.1 or later.
  • The `ExperimentResult` class' `rows` and `summary_evaluations` attributes are deprecated and will be removed in the next major release. These attributes will only store the results of the first run iteration for multi-run experiments. Use the `ExperimentResult.runs` attribute instead to access experiment results and summary evaluations.