●Change8

v3.10.0

📦 datadog-sdkView on GitHub →
✨ 19 features🐛 24 fixes⚡ 2 deprecations🔧 38 symbols

Summary

This release introduces significant new features across LLM Observability, Dynamic Instrumentation, and CI Visibility, alongside numerous bug fixes for tracing stability and generator handling. Several environment variables and method arguments have been deprecated.

✨ New Features

  • DSM: Added flag in set_consume_checkpoint() to indicate if DSM checkpoint was manually set.
  • Tracing: Added environment variable `DD_RUNTIME_METRICS_RUNTIME_ID_ENABLED` to enable runtime metrics tagging with the current runtime ID, replacing `DD_TRACE_EXPERIMENTAL_RUNTIME_ID_ENABLED`.
  • Tracing: Added support for Azure Service Bus producers in `azure_servicebus` integration.
  • Tracing: Added messaging span attributes for service bus triggers in `azure_functions` integration.
  • Tracing: Added distributed tracing support for Service Bus triggers in `azure_functions` integration.
  • Tracing: Added patching of `ddtrace_api.tracer.set_tags` to the `ddtrace_api` integration.
  • Logging: Enabled trace-log correlation by default for structured loggers (loguru, structlog, logbook).
  • Logging: Added support for trace-log correlation via remote configuration for structured loggers.
  • Dynamic Instrumentation: Code Origins for Spans is now automatically enabled when Dynamic Instrumentation is turned on.
  • LLM Observability: Introduced tracing support for bedrock-agent-runtime `invoke_agent` calls, converting internal bedrock traces to LLM Observability spans.
  • LLM Observability: Added support for configuring proxy URLs using `DD_LLMOBS_INSTRUMENTED_PROXY_URLS` or `instrumented_proxy_urls` argument; spans sent to a proxy URL now appear as workflow spans.
  • LLM Observability: Added tracing support for the OpenAI Responses endpoint.
  • LLM Observability: Introduced tracing support for Google's Generative AI SDK for Python's `generate_content` and `generate_content_stream` methods (`google_genai`).
  • LLM Observability: Introduced tracing support for PydanticAI's `Agent.iter` and `Tool.run` methods (`pydantic_ai`).
  • CI Visibility: Preliminary support introduced to link children pytest-xdist tests (and related entities) to their parent sessions.
  • Exception Replay: Added in-product enablement support.
  • Code Security (IAST): Added handling for IAST security controls custom validation and sanitization methods.
  • Profiling: Added gevent support to the new stack sampling mechanism (stack v2).
  • AAP: Introduced the WAF trace tagging feature to gather API security findings without generating appsec events.

🐛 Bug Fixes

  • Tracing: Fixed dangling reference issue in algoliasearch integration.
  • Tracing: Resolved a `TypeError` encountered when truncating a large bytes object during encoding.
  • Tracing: Resolved an issue where agent-based sampling rates were not correctly applied after process forking or tracer reconfiguration.
  • Tracing: Resolved a bug where `os.system` or `subprocess.Popen` could return the wrong exception type.
  • Tracing: Fixed an issue where traced nested generator functions had their execution order subtly changed, affecting stack unwinding during exception handling.
  • Tracing: Fixed an issue where the `@tracer.wrap()` decorator failed to preserve return values from generator functions, resulting in `StopIteration.value` being `None`.
  • Tracing: Enabled parsing of distributed tracing metadata in `rq` perform job.
  • AAP: Resolved an issue where `track_user` was generating additional unexpected security activity.
  • AAP: Resolved an issue where the new ATO SDK `track_user` was reporting different email, name, scope, and role for the tracked user.
  • CI Visibility: Resolved an issue where test spans would be left unfinished if the `pytest_runtest_protocol` hook was overridden in `conftest.py`.
  • CI Visibility: Resolved an issue where code coverage would not be enabled if ddtrace was enabled via the `PYTEST_ADDOPTS` environment variable.
  • azure_functions: Resolved an issue causing functions to throw an error on loading when the `function_name` decorator followed another decorator.
  • LLM Observability: Fixed an issue where modifying bedrock converse streamed chunks caused traced spans to show modified content.
  • LLM Observability: Resolved an issue where manual instrumentation raised `DD_LLMOBS_ML_APP` missing errors when LLM Observability was disabled.
  • LLM Observability: Fixed an out of bounds error in litellm when handling streamed responses due to missing keyword arguments for the number of choices.
  • LLM Observability: Fixed an issue where the trace ID exported from `export_span` was incorrect.
  • LLM Observability: Resolved an `AttributeError` in `langchain` that could occur when async tasks were cancelled during `agenerate` calls.
  • Dynamic Instrumentation: Fixed an incompatibility issue with code origin that caused line probes on entry point functions to fail instrumentation.
  • Dynamic Instrumentation: Fixed an issue with instrumentation of the first line of an iteration block (e.g., for loops) that could cause undefined behavior.
  • Dynamic Instrumentation: Fixed an issue preventing line probes from being instrumented on a line containing only `try:` for CPython 3.11+.
  • Dynamic Instrumentation: Fixed an issue with the instrumentation of generators using Python 3.10.
  • Code Origin: Fixed a potential memory leak when collecting entry span location information.
  • Logging: Ensured `ddtrace.tracer.get_log_correlation_context()` returns prefixed attributes (e.g., `dd.trace_id`) aligning with documentation.
  • Logging: Fixed an issue where `ddtrace.tracer.get_log_correlation_context()` returned the global service name instead of the current span's service name.

🔧 Affected Symbols

DD_DYNAMIC_INSTRUMENTATION_UPLOAD_FLUSH_INTERVALDD_DYNAMIC_INSTRUMENTATION_UPLOAD_INTERVAL_SECONDSrecord_exceptionset_consume_checkpointDD_RUNTIME_METRICS_RUNTIME_ID_ENABLEDDD_TRACE_EXPERIMENTAL_RUNTIME_ID_ENABLEDazure_servicebusazure_functionsddtrace_api.tracer.set_tagslogurustructloglogbookddtrace_apibedrock-agent-runtime invoke_agentDD_LLMOBS_INSTRUMENTED_PROXY_URLSinstrumented_proxy_urlsOpenAI Responses endpointgoogle_genai generate_contentgoogle_genai generate_content_streampydantic_ai Agent.iterpydantic_ai Tool.runpytest-xdistpytest_runtest_protocolconftest.pyPYTEST_ADDOPTSfunction_name decoratorbedrock converse streamed chunksDD_LLMOBS_ML_APPlitellm streamed responsesexport_spanlangchain agenerateos.systemsubprocess.Popentracer.wrapalgoliasearchrq perform jobtrack_userddtrace.tracer.get_log_correlation_context

⚡ Deprecations

  • The environment variable `DD_DYNAMIC_INSTRUMENTATION_UPLOAD_FLUSH_INTERVAL` is deprecated in favor of `DD_DYNAMIC_INSTRUMENTATION_UPLOAD_INTERVAL_SECONDS`.
  • The `escaped` and `timestamp` arguments in the `record_exception` method are deprecated and will be removed in version 4.0.0.