← back
LLM Observability, Evaluation, Experimentation Platform — Dat Ngo, Arize
Takeaway
Combine multi-level observability with calibrated evaluation signals to improve agents without overlooking regressions.
Summary
- Arize uses OpenTelemetry instrumentation to capture traces and spans as records of what nondeterministic agents actually did.
- Session views track conversations and state across runs, while aggregate trajectory views reveal branch frequencies, loops, latency, and tool-ordering failures.
- Evaluation signals include LLM judges, human feedback, trusted golden datasets, deterministic checks, and business outcomes such as revenue, savings, or time saved.
- Golden datasets calibrate judges against domain expertise, and schema validation provides cheap deterministic checks; experiments must detect regressions introduced by apparent fixes.
observabilityopentelemetryllm-as-judge
Original description
Your agent called tool B before tool A, and B has a dependency on A. You did not catch it because nothing in your code audits agents. The telemetry does. Dat from Arize AI walks through what observability actually means when the system you are debugging is nondeterministic and the execution path changes with every run. The talk covers the five flavors of eval signal (LLM as judge, human feedback, golden datasets, deterministic checks, business metrics), what scope to run them at (single span, multispan, trajectory, session), and where this is heading. Arize Phoenix is open source, runs as a single container, no Kubernetes required. The enterprise product adds an AI layer called Alex that scans traces, surfaces high latency and errors, and creates evals automatically. The stated goal: automate you out of the observability loop entirely. Speaker info: / datdarylngo https://x.com/dat_attacked