← back
How agent o11y differs from traditional o11y — Phil Hetzel, Braintrust
Takeaway
Agent observability must measure behavioral quality and handle large text-rich traces alongside conventional reliability metrics.
Summary
- Traditional observability measures uptime, errors, and latency; agent observability must also assess grounding, tool choices, and brand alignment.
- Nondeterministic agent paths make execution quality a separate question from whether infrastructure is operational.
- Braintrust has encountered traces exceeding a gigabyte and individual spans around 20 MB, creating ingestion and query challenges for semi-structured text.
- The system must support both near-real-time inspection and programmatic trace queries used by automated improvement workflows.
observabilitybraintrustagent-quality
Original description
Traditional observability answers one question: is the system up? Phil Hetzel from Braintrust argues that question is not the right one for agents. An individual agent trace can exceed a gigabyte. A single span can hit 20 megabytes. The data is semistructured, packed with unstructured text, and still arrives in real time. None of the systems built for uptime monitoring were designed to ingest, index, and actually use that. Braintrust built a custom database from scratch for this problem: a write ahead log for instant visibility, analytical indexes for fast filtering, and a forked version of Tantivy (a Rust based full text search library similar to Apache Lucene) so an engineer can query every trace that mentioned a specific word. The other difference is who does this work: clinicians, lawyers, and wealth advisers now open traces directly to grade whether an agent responded correctly, and their written justifications become the training signal for automated scoring functions. The human annotations surface the failure modes. The scoring functions scale them. Speaker info: / philliphetzel