← back
Your Agents Need a Save Button - Hamza Tahir, ZenML
Takeaway
Checkpointing executable agent state turns production observations into repeatable experiments on models, tools, and policies.
Summary
- Ordinary traces record tool inputs and outputs but often omit executable state, code, filesystem artifacts, and runtime context needed for faithful replay.
- ZenML's Kitaru connects durable checkpoints to traces so teams can resume from an intermediate point and test alternative decisions.
- A support-agent demo replays a run with GPT-5 Nano, then separately mocks a policy tool, reusing earlier checkpoints and comparing resulting branches.
- The proposed improvement loop selects production runs, replays a change, diffs behavior and cost, and uses the evidence to decide what to ship.
checkpointingagent-replaykitaru
Original description
Most of an agent's life is spent waiting - on a tool, a human, the next step - and the whole time you're holding a live process awake and billing for it. Multiply that across every agent your org wants to run overnight and the math stops working. A save button fixes the obvious stuff: freeze an agent to durable state, drop its compute to zero, bring it back in milliseconds when there's work, and if it crashes, resume from the last save instead of re-burning every token from the top. The interesting part is what a save button does after the run is over. Reload an agent to any point in its trajectory, change one thing, a prompt, a model, a tool, and watch whether it does better. A finished run stops being a log you read and becomes something you re-run and improve. I'll argue that this one primitive, a checkpoint, is the most underrated thing in agent infrastructure, and follow it down to the sandboxes and Kubernetes-shaped infra the industry is quietly racing to build so a million saved agents can sleep for free. You'll leave with a model for running and improving agents at scale without paying to keep them awake. Speakers: Hamza Tahir (ZenML): Hamza Tahir is co-founder and CTO of ZenML and co-founder of Kitaru, the durable runtime for AI agents. He's spent a decade building production ML and AI infrastructure used by JetBrains, the German Bundeswehr, and Adeo, and writes and speaks on what actually breaks when agents hit production. X/Twitter: https://x.com/htahir111 LinkedIn: / hamzatahirofficial GitHub: http://github.com/htahir1