Incident Triage

The incident fires. The investigation starts itself.

When the vehicle-to-cloud pipeline starts throwing errors, telemetry ingestion degrading, a consumer group falling behind, a service timing out after a deploy, the alert itself opens an investigation. A policy-governed agent gathers the evidence a responder would have hunted for by hand, and posts a diagnosis that cites all of it.

Where the evidence comes from · one investigation

14
Metrics · dashboards & panels 6
Logs & traces 4
Deploy annotations 3
Stream health · consumer lag 1

What the agent gathers

Four evidence surfaces, one investigation

Metrics

Error rates, latency percentiles, saturation, queried from your dashboards through a governed tool path, captured as artifacts.

Logs & traces

The distributed trace that shows which downstream dependency is timing out, and the log lines that say why.

Deployment annotations

What shipped, and when, correlated against the moment the error rate moved. Most incidents start with a deploy.

Stream health

Consumer lag, partition skew, and backpressure across the messaging layer that carries vehicle telemetry into the platform.

Trigger to review

The full lifecycle of an investigation

01 · TRIGGER

The incident event opens the investigation

An alert fires, an incident is declared in your on-call tooling, or an operator dispatches an event by hand. Each is an event, not a chat prompt, and each resolves to a configured agent, tool set, and policy before any work begins.

  • Alert-shaped, incident-shaped, and operator-dispatched triggers
  • Agent and policy resolved from configuration
  • The provenance ledger opens with the investigation
investigation · INC-4212

telemetry-ingest · elevated errors

Alert · error rate 14× baseline since 03:02

Opened
Agentincident-triageResolved
Policyobservability-readonlyEnforced
02 · INVESTIGATE

Evidence collected live, under policy

The agent works the incident the way a strong responder would: correlate the error spike with the deploy timeline, follow the trace to the dependency that's timing out, check whether the streams behind it are healthy. Every tool call runs through the governed registry, and every artifact is preserved at the moment of collection.

  • Read-only observability access by default
  • Artifacts captured and bound to the investigation
  • Long-running: context holds across the whole incident
evidence.ts
// every call is governed and recorded
await run.tools.grafana.query({
  panel: "5xx-by-route",
  window: "03:00..03:15"
})
→ artifact ev-07 · captured · cited later

await run.tools.k8s.rollouts("telemetry-ingest")
→ artifact ev-08 · deploy at 03:02
03 · REPORT

A cited diagnosis, reviewed by a human

The report lands where your responders already work. A deploy at 03:02 introduced an aggressive timeout on the battery-state dependency; the 5xx spike follows it by seconds; here are the three artifacts that show it. The on-call engineer confirms or challenges, and the whole exchange joins the provenance trail.

  • Every finding cites the artifact behind it
  • Report links to the transcript and provenance trail
  • Nothing is acted on without human review
report · posted
Diagnosis · timeout on battery-state cites ev-07..09
Next step · raise timeout, redeploy proposed
On-call review · confirmed recorded

Triage, accounted for

What a governed investigation guarantees

Report claims backed by cited evidence

100%
Cited to a captured artifact 100%
Uncited assertions 0%

Tool calls inside approved policy

100%
Resolved through the registry 100%
Executed outside policy 0%

Steps reconstructable from the trail

all
On the provenance ledger 100%
Reconstructed from memory 0%

Give every incident an investigation with receipts.

Open the investigation on the alert itself, gather evidence under policy, and hand your responders a diagnosis they can verify claim by claim.