v4.12.0

✨ New Features

• AI Guard: Introduced `DD_AI_GUARD_OPENAI_ENABLED` environment variable (defaults to `true`) as a per-provider kill switch for OpenAI SDK auto-instrumentation.
• AI Guard: Added support for LangChain 1.0+ agents built with `create_agent`.
• AI Guard: Added opt-in evaluation of streamed OpenAI Chat Completions and Responses via `DD_AI_GUARD_ANALYZE_STREAM_RESPONSES_ENABLED` (defaults to `false`). Streams are buffered and evaluated before delivery, raising `AIGuardAbortError` on a block.
• AI Guard: Introduced standalone mode. When APM tracing is disabled (`DD_APM_TRACING_ENABLED=false`), AI Guard traces are still sent to Datadog with `USER_KEEP` sampling priority and the AI Guard decision maker, attributed to AI Guard without triggering APM host billing.
• AI Guard: Added opt-in buffered stream response evaluation for Anthropic streaming calls when `DD_AI_GUARD_ANALYZE_STREAM_RESPONSES_ENABLED=true`. Streams are buffered, the full response is evaluated, and chunks are replayed after an ALLOW verdict. On DENY/ABORT, `AIGuardAbortError` is raised.
• AAP: Added normalized HTTP route span tag (`_dd.appsec.normalized_route`) for Tornado, following RFC-1103 and existing implementations for FastAPI, Starlette, Django, and Flask. The tag is emitted on requests with `http.route` when API Security is active. Tornado routes with named capturing groups produce named parameters, and positional groups produce auto-numbered placeholders. Optional trailing-slash patterns are treated as not declaring a trailing slash.
• AppSec: Detects OpenAI LLM usage within API endpoints. When AppSec is enabled and an OpenAI `chat.completions`, `completions`, or `responses` call is made during a web request, the WAF address `server.business_logic.llm.event` is emitted with provider and model name, enabling WAF rules to tag traces with `appsec.events.llm.call.provider` and `appsec.events.llm.call.model`. Supports sync/async OpenAI clients, Azure OpenAI, and streaming. Only the first LLM call per request is recorded.
• aws_durable_execution_sdk_python: Added `aws.durable.operation_attempt` tag to `aws.durable.step` and `aws.durable.wait_for_condition` spans, indicating the attempt number (0 for original, 1+ for retries).
• ray_serve: Added instrumentation support for `ray[serve]>=2.47.1`. Traces Serve HTTP/gRPC requests, deployment routing/execution, and `DeploymentHandle.remote()` calls, with distributed trace context propagation. See [Ray](https://ddtrace.readthedocs.io/en/stable/integrations.html#ray) documentation.
• LLM Observability: Supports attaching audio to messages on LLM spans via an optional `audio_parts` field in `LLMObs.annotate`, allowing collection and rendering of audio data.
• LLM Observability: Added `sample_rate` argument to `LLMObs.enable()` to configure the proportion of LLMObs traces to sample in code, taking precedence over `DD_LLMOBS_SAMPLE_RATE`.
• LLM Observability: Added `SyncExperiment.rerun_evaluators()` to re-run evaluators on stored task outputs without re-executing the task function. Reads from `experiment.result` and returns a new `SyncExperiment` with fresh evaluations, preserving span IDs and trace IDs via replay span copies. `missing_task_strategy` controls behavior for rows with errors.
• LLM Observability: LLM span data can now be exported to the Datadog Agent over the APM trace and is reliably delivered even when the APM trace is not sampled.
• LLM Observability: OpenAI integration captures input and output audio on audio chat completions as `audio_parts` on LLM span messages. Audio in streamed responses is not yet captured.
• LLM Observability: Added support for `DD_LLMOBS_SAMPLE_RATE` to control the proportion of LLM Observability spans retained (defaults to 1.0).
• LLM Observability: Added span `metadata` to data exposed to span processors registered with `LLMObs.enable(span_processor=...)` or `LLMObs.register_processor(...)`, allowing processors to read and modify metadata.
• LLM Observability: Added support for submitting LLM Observability spans for `mistralai` client chat and embedding calls.
• profiling: Implemented heap live count data collection.
• profiling: Revamped adaptive sampling mechanism for better workload matching.
• profiling: PyTorch profiler now properly shows nested frames in flame graphs.
• mistralai: Added tracing support for `mistralai` client chat and embedding calls.
• pytorch: Added a `pytorch.rank` lifetime span for PyTorch distributed training, open from `init_process_group` to `destroy_process_group` or process exit. Includes tags like `rank`, `world_size`, `framework`, `launcher`, `torch.distributed.backend`, and `training_job_id`. When running under Ray Train, `ray.train.run_name`, `ray.submission_id`, and `ray.metadata.*` are also applied. Enable with `DD_PATCH_MODULES=pytorch:true`.
• Added ability to override manually installed tracer for Single Step Instrumentation. `DD_INJECT_EXPERIMENTAL_OVERRIDE_USER_DDTRACE=true` can be set to ensure injected libraries take precedence.
• CI Visibility: The `DD_TEST_MANAGEMENT_ATTEMPT_TO_FIX_RETRIES` environment variable is now respected by the `ddtrace.testing` internal pytest plugin, overriding the retry count returned by the Datadog API for 'attempt to fix' tests.