← back
Using LLMs to Secure Source Code — Eugene Yan, Anthropic
Takeaway
Effective LLM-assisted security depends on explicit system context and reproducible exploit verification so teams can prioritize and patch real vulnerabilities.
Summary
- Eugene Yan argues that stronger models are shifting the security bottleneck from vulnerability discovery toward verification, triage, and patching; he cites Anthropic scanning over 1,000 open-source repositories and producing 23,000 candidates.
- The recommended workflow has two setup steps, threat modeling and sandbox construction, followed by a repeating discovery, verification, triage, and patching loop.
- Threat models should record assets, entry points, trust boundaries, and compensating controls; code alone misses operational context and design assumptions held by system experts.
- Models can bootstrap threat models from documentation, code, past patches, and vulnerability records, then interview experts to clarify undocumented risks and exclusions.
- Representative, reproducible sandboxes let agents execute proof-of-concept exploits to reject speculation; the proposed isolation uses VMs without network egress or cloud credentials.
llm-securitythreat-modelingvulnerability-verification
Original description
Mozilla shipped about 20 security fixes a month across Firefox in early 2025. In April it shipped 400, a 20x jump, and it credited roughly two thirds of them to a frontier model. That is the shift Eugene Yan came to describe: models are now finding and fixing real vulnerabilities at scale. Anthropic's own scan of more than a thousand open source repos surfaced 6,200 high or critical issues out of 23,000 candidates, reported 1,600 to maintainers, and saw about 100 patched upstream. Finding bugs, it turns out, is no longer the hard part. The bottleneck has moved to verifying, triaging, and patching them. The talk walks a six step workflow through one running example: a five line order lookup with a SQL injection hiding in a Python string. The two setup steps are a threat model and a sandbox. A written threat model alone pushes the true positive rate to 90%, because a model has great context of the code but poor context of the system, all the design decisions that live only in someone's head. The four loop steps read like a machine learning pipeline: discovery optimizes for recall, then a separate verification agent, kept independent and adversarial so it never sees the discovery reasoning, optimizes for precision by detonating the exploit in a fresh container. Triage protects the scarcest resource, engineer attention, and patching closes the loop so the same bug cannot return. His parting advice: start this week on open source dependencies, keep your hands on the wheel before automating, and remember that scanning was never the bottleneck. Speaker info: https://x.com/eugeneyan https://github.com/eugeneyan https://eugeneyan.com Timestamps: 0:00 - Working with security teams to find and fix vulnerabilities 0:49 - Three trends in model security capability 1:16 - Cybersecurity benchmarks and the step jump in capability 1:54 - Mozilla's 20x jump in monthly security fixes 2:44 - Log4Shell, Heartbleed, and why this matters 3:22 - Anthropic's scan of a thousand open source repos 3:35 - The bottleneck shifts to verify, triage, and patch 3:48 - Why agentic harnesses changed the game 4:29 - The six step workflow 5:31 - A running example: the order service 5:45 - Step 1: the threat model and 90% true positives 7:42 - Step 2: the sandbox for isolation and reproducibility 9:24 - Step 3: discovery and the five line SQL injection 11:44 - Step 4: independent adversarial verification 13:36 - Step 5: triage and the scarcity of engineer attention 15:52 - Step 6: patching and closing the loop 17:19 - It all looks like a machine learning pipeline 17:43 - The non technical bottlenecks are harder 18:47 - Organizational bottlenecks: routing, severity, bandwidth 20:05 - Three takeaways and how to start this week "The bottleneck has now shifted to verification, triage, and patching." (3:39) "A model has great context of the code but poor context of the system." (6:06) "Things that can be solved with money are not really problems. But human attention doesn't scale." (18:14) "Scanning was never the bottleneck." (20:38)