← back
Wearing the Agent: From Group Chats to Glasses — Sai Krishna Rallabandi
Takeaway
Group and wearable agents must manage who receives information and enforce action boundaries across a shared, evolving context.
Summary
- The Judith agent has operated among friends and family for eight months, exploring the shift from single-user assistants to group and wearable interactions.
- Examples route sensitive replies through direct messages or glasses instead of shared channels, coordinate calendars, and curate evolving group memory.
- Group agents encounter adversarial content through messages, screenshots, emails, and web pages; individually benign skills can combine into unsafe information flows.
- The proposed security direction places fast deterministic checks at action boundaries, including secret access and outbound data movement.
group-agentswearablesprivacy
Original description
Almost every agent today is built for one user, and Sai Krishna Rallabandi has spent about eight months on what breaks when you drop one into a group chat instead. Running a personal agent across a real group, and eventually onto glasses, forces two hard problems. The first is memory: conversations that last for days across several people, where the agent has to track who said what and surface only what is relevant without bloating its context. The second is security, and it is where the talk digs in. He points to a finding that two skills, each safe on its own, can collide once they share infrastructure, so a reporting agent leaks information at low frequency. His defense is to have an agent read everything and design a guard, then fine tune a small model with a per user adapter that only releases information appropriate to the context and catches prompt injection that regex would miss. The through line is that group and wearable settings break the single user assumptions most agent harnesses are built on, so memory and security have to be redesigned around the group. Speaker info: https://x.com/Saikallis9012 / sai-krishna-rallabandi-8595418b https://saikrishnarallabandi.github.io/ Timestamps: 0:00 - Introduction: agents for groups, not one user 1:28 - Why single user agents fall short 2:45 - Eight months in a real group chat 3:47 - What changes when the agent joins a group 5:17 - Two problems: guarding and memory 5:55 - Securing an agentic system 7:44 - When two safe skills collide 9:27 - Designing a guard agent 10:21 - A small model with per user adapters 11:49 - Catching prompt injection 12:29 - Designing memory for groups 15:29 - Context growth and token cost