The judge rewards a polished answer while the underlying task remains unfinished.
Fast, readable scoring protects review speed, while trajectory-aware scoring protects confidence in production quality.
Who you’d be doing this for
“I opened three green traces today, and every one had stopped after the lookup failed.”
Grete Lindqvist · AI Application Engineer
Runs a production support agent and uses trace-level evaluations to decide which failures require investigation.
What is at stake
Passing scores are labeling incomplete agent answers as healthy after tool calls return no usable result. You must weigh stronger failure detection against false alarms, latency, and evaluation cost.
Why it isn’t already fixed
Every obvious fix costs something else. That’s the part you’d have to decide.
- polished answers vs completed tasks
- failure recall vs false alarms
- richer trace context vs evaluation cost
- local responsiveness vs shared rubric consistency
Why Datadog
LLM Observability links evaluation results to agent traces, so a misleading pass score obscures the tool failure customers need to debug.
Written with these in mind
Not your kind of problem? 40 more at Datadog, or browse every organization.
This is the setup. The work is inside.
Running it puts you in the room: the full situation and its constraints, stakeholders who push back in their own words, and the decisions that are yours to make. What you produce becomes a Day One Plan — work you can show someone instead of describing.