← back
Fighting AI with AI — Lawrence Jones, Incident
Takeaway
Make evaluation and debugging data accessible through focused tools and files so AI can help maintain complex AI systems.
Summary
- Incident.io's AI SRE investigates telemetry and code through hundreds of queries and prompts, making manual quality assessment difficult to scale.
- Prompt evaluations live in YAML beside Go code, but full production incident examples can make test files huge and hard for coding agents to edit.
- A small eval CLI exposes listing, editing, replacing, and adding cases without loading entire suites into context, enabling agent-readable runbooks.
- The team also advocates exporting debugging interfaces as downloadable file systems and using agents for repeatable analysis pipelines.
ai-sreeval-toolingagent-observability
Original description
Incident's AI SRE runs hundreds of prompts per investigation across logs, metrics, traces, and code. When it produces a wrong root cause analysis, there is no tractable way for a human to read through the full trace and find where the reasoning went sideways. Lawrence Jones, founding engineer at Incident.io, describes the moment the team realized they needed AI to debug their AI. The talk covers three patterns they built. A small CLI lets coding agents read and edit eval YAML files that had grown too large for agents to work with directly, enabling a red-green runbook where the agent writes a failing eval, fixes the prompt, and checks nothing else broke. Their bigger unlock was serializing every UI debugging view as a downloadable file system: drop it into a Claude Code session, describe the bad behavior, and the agent traces through the prompt hierarchy to tell you exactly which prompt to change. For fleet-scale analysis, 25 agents run in parallel each analyzing one investigation, then a second stage clusters the results to surface systemic failure patterns across customer accounts. Speaker info: https://x.com/lawrjones / lawrence2jones