← back
Everything Is a Rollout — Alex Shaw + Ryan Marten, Terminal-Bench, Harbor, Laude Institute
Takeaway
Evaluate and improve agents through repeatable environment rollouts with explicit verification and preserved trajectories.
Summary
- Harbor treats agent development as empirical machine learning: environments serve as data, prompts and tools act like tunable parameters, and reward hacking resembles overfitting.
- An evaluation environment combines a task instruction, a sandbox where the agent acts, and a verifier that checks the resulting state.
- Harbor packages environments as interoperable directories and supports agents running either inside or outside the sandbox.
- A rollout starts the environment, executes the agent until a stopping condition, records its trajectory, runs verification, and tears down the sandbox.
harboragent-evaluationrl-environments
Original description
Alex Shaw and Ryan Marten present a rollout-centered view of evaluating and improving AI agents. Drawing on their work on Harbor, Terminal-Bench, and OpenThoughts-Agent, they connect sandboxed environments, agent evaluations, and optimization workflows into a practical framework for generating and learning from rollouts. Speakers: Alex Shaw — Member of Technical Staff, Laude Institute Alex is the creator of Harbor, a framework for evaluating and optimizing agents and language models in sandboxed environments. / alexgshaw Ryan Marten — Member of Technical Staff, Laude Institute Ryan builds Harbor and works on research-to-production efforts including Terminal-Bench and OpenThoughts-Agent. / ryan-marten Harbor: https://www.harborframework.com/ GitHub: https://github.com/harbor-framework/h...