← back
We Vetted 2000 AI Skills Before They Reached Developers — Lucas Palma, Nubank
Takeaway
Secure the instructions and tools shaping AI development workflows before distributing them to engineers.
Summary
- Nubank treats agent skills, plugins, MCP servers, and rules as software supply-chain dependencies because they influence developer actions and generated code.
- An internal marketplace gates skill publication through local scans and CI review combining deterministic checks with contextual LLM analysis.
- Findings appear in pull requests and SARIF reports, enabling remediation, blocking decisions, and integration with vulnerability management.
- The initial review covered more than 2,000 skills and identified over 1,500 risks, with roughly 1,000 promptly remediated; checks included destructive commands, exposed credentials, and excessive permissions.
skillssupply-chain-securitynubank
Original description
An AI skill is a piece of code you hand a model to extend what it can do, and once engineers start sharing skills with each other, each one becomes a supply chain risk, more so inside a regulated bank. Lucas Palma's security team at Nubank built Skill Vector to sit between a skill and the internal marketplace, so nothing reaches developers unvetted. Every skill is scanned first with deterministic checks, for destructive shell commands and the like, then with an LLM for the context those checks miss, and only then does it get a decision and permissions scoped to who will use it. Running that gate over more than 2,000 skills surfaced real problems, since a single skill can carry many, and fed them into the bank's vulnerability management program with approval gates and human confirmation. What worked was pairing deterministic scans with LLM review; what needed work was the guidance the system gave and the habit of running skills locally before vetting. The lesson he leaves is simple: treat skills like any other dependency, and only what clears the gate belongs in the marketplace. Speaker info: / lucaspalma Timestamps: 0:00 - Introduction: making code safe at a bank 1:32 - AI skills as a supply chain risk 2:50 - What counts as an AI skill 3:57 - The extra weight of a regulated environment 6:07 - From plugins to a vetted marketplace 6:58 - What Skill Vector does 7:37 - Deterministic checks, then the LLM 10:00 - Scanning over two thousand skills 11:22 - What worked and what needed improvement 13:30 - Approval gates and human confirmation 14:23 - Next steps and policies