Use case · Audit & evaluation
Whether an agent "works" can't be judged from its final answer, a wrong process can still produce a plausible-sounding conclusion. Knowing takes the full record: what it did, which evidence it used, which tools it called, what it ignored, and where a human intervened. Sentinel produces that record as a side effect of running.
Deploy introduces timeout on battery-state dependency
What the trail unlocks
Every tool call and output, every model prompt and response, every artifact and decision, in order, on an append-only ledger. The unit of evaluation is the whole investigation.
Seed an incident with a known root cause and check whether the agent found the right evidence, not whether its answer merely sounds right. Grade process, not prose.
An auditor can replay any investigation as one causal story, trigger to recommendation, long after everyone involved has forgotten the details. No silent edits, ever.
Where a person confirmed, challenged, or took over is part of the record. That's how you measure trust honestly, including the cases where the agent needed help.
The method
Dispatch a realistic incident, alert-shaped or incident-shaped, against seeded observability data where the root cause is known in advance. The agent investigates it exactly as it would a live incident.
// dispatch a ground-truth incident await sentinel.events.dispatch({ sample: "ingest-5xx-after-deploy", truth: { cause: "timeout", evidence: ["deploy", "trace"] } }) → investigation runs as if live
Did it find the deploy? Did it pull the right trace? What did it skip? Because the trail records everything, including omissions, the gaps are as visible as the hits, and they become the next iteration's fix.
Ground-truth scenarios, full execution traces, and a record that answers the hard questions long after the incident is closed.