Datadog Brings Production Experimentation to AI Agent Observability

Nov 17, 20263 min read
Datadog is adding a new feature called Online Experiments to its Agent Observability platform. This lets engineering teams compare different versions of AI agents using real user traffic instead of relying only on prepared test data. The feature combines agent evaluation scores with information about how the system runs, such as response time, token use, execution cost, and errors. This helps teams find out whether changes to models, prompts, information retrieval methods, or tools really improve an agent.
The announcement addresses a growing problem in AI agent development: a change that works well in a controlled test may behave very differently when real users make unpredictable requests. Datadog brings controlled experiments, production performance data, and records of agent activity together. This helps teams understand both the results and the actions that led to them.
Offline evaluations are still an important part of developing AI agents. Teams can test different versions using prepared datasets, check their responses, and find known problems before releasing them.
However, these tests are only as good as the data and scoring rules behind them. New ways of using a product, unexpected user requests, and changes in how tools respond can reveal problems that existing tests never covered.
Online experiments help solve this problem by testing agent behavior during real user interactions. Teams can compare an unchanged version with a modified version by randomly sending production traffic to each one. This reduces the effect of differences between user requests.
This gives teams a more realistic picture of how an agent works under actual conditions. It can also help them find new situations that should later be added to their offline test datasets to check future changes.
Datadog connects its Agent Observability evaluation tools with its Experiments feature. This lets teams compare evaluation scores with standard measurements of system performance.
Teams can measure response quality, accuracy, and completeness while also tracking how long tasks take, how many tokens they use, their estimated cost, and workflow errors. Feature Flags control how traffic is divided. This allows engineers to show a change to a small percentage of users before making it available to more people.
This matters because improvements often come with trade-offs. A cheaper model might need more attempts to finish a task. A changed method for finding information might provide better evidence but take more time to run.
Datadog also connects experiment results to individual agent traces. Engineers can examine tool calls, the information sent to tools, and the results returned. This helps them understand why one version behaved differently from another.
The article describes an experiment that compared GPT-5.6 Luna with GPT-6 Luna, giving each model half of the traffic. An evaluator based on a large language model (LLM) gave investigation quality a score from one to five. It looked at accuracy, completeness, and whether conclusions were supported by the available evidence.
The reported results showed a 44.3% reduction in estimated costs for the newer model. The quality scores remained very close, at 3.84 and 3.86, respectively.
However, the system performance results raised some possible concerns. The newer model appeared to increase execution time and workflow errors. But the confidence intervals were too wide to show that these problems had definitely become worse.
Looking at the activity records showed that the newer model also provided incorrect tool parameters more often. It usually recovered after trying again. The overall quality score did not clearly show this behavior.
Datadog used this finding to improve how tools were handled before moving ahead with the model change. The failed calls also provided examples that could be added to future offline regression tests.
Datadog is not the only company working on this problem. Tools such as LangSmith and Langfuse offer ways to track and evaluate AI agents. Experiment platforms such as LaunchDarkly help teams control how new features are released. Datadog connects these activities through its existing observability platform, linking evaluation results directly to production experiments and records of agent activity.
Datadog's broader idea is that online experiments can create a feedback loop between what happens in production and what teams test during development. Unexpected failures can become new regression test cases. Later experiments can then show whether changes have actually fixed those problems.
The main change is in the question teams ask. Instead of only asking whether an agent passes a prepared test, they can find out whether it actually works better in real situations. For teams building AI agents, testing with real production traffic may become just as important as traditional software testing.
