← back
Harnesses in AI: A Deep Dive — Tejas Kumar, IBM
Takeaway
Reliable agents depend on the runtime, controls, and verification around the model.
Summary
- An agent harness comprises the surrounding runtime that grounds a model in a controlled environment; it differs from a machine-learning evaluation harness.
- Core components include a tool registry, model access, context management, guardrails, the agent loop, and a verification step.
- Coding harnesses can compact context, cap tool calls, and run lint or tests to check completed work.
- A browser-agent demonstration uses GPT-3.5 Turbo and Playwright to explore improving behavior through harness design while keeping the prompt fixed.
agent-harnessesplaywrightreliability
Original description
Tejas will be back on stage at the World's Fair next week! see https://ai.engineer/wf and use YOUTUBEPROMO for new tickets only. Join 6000 AI engineers at the "Superbowl of AI"! --- The agent hit a login page, panicked, reported success anyway, and the upvote never happened. Tejas Kumar's diagnosis: not a prompt problem. A harness problem. The demo builds a browser agent on GPT-3.5 Turbo (consciously choosing a VERY old model to show how good harness eng can improve it a lot) against Hacker News and layers in a harness without touching the prompt once. Guardrails cap iterations and compact context. A verify step reads the tool call history to catch the agent lying about what it did. A login handler watches the browser URL each loop and injects credentials programmatically when it hits the login page. By the end the cheap old model reliably logs in and upvotes the post. Speaker info: https://x.com/TejasKumar_ / tejasq https://github.com/TejasQ Timestamps: 0:00 Introduction to Tejas Kumar and AI Harnesses 1:45 Why we use harnesses: Reliability and control 3:00 Defining an agent harness from first principles 4:32 Key components of an agent harness (Tooling, Context, Guardrails) 5:59 Starting the demo: Building a browser agent 7:00 Inspecting the initial agent loop 8:12 The problem: Agent failure and hallucination 10:20 Adding guardrails and context management 11:54 Refactoring into a formal harness 13:02 Implementing a verify step to catch lies 15:36 Implementing a login handler for programmatic access 17:42 Final demonstration: Successful autonomous upvoting 18:34 Summary and the future of dynamic harnesses