← back
Run Frontier AI at Home — Alex Cheema, EXO Labs
Takeaway
Making frontier models practical locally requires optimizing memory-bound inference throughout the stack rather than copying training-oriented infrastructure.
Summary
- EXO works across models, software, and hardware to reduce the cost of running capable open models locally and give users more control over data and availability.
- Cheema argues that hardware and research optimized for training FLOPs overlook opportunities in low-batch inference, where memory movement is often the limiting factor.
- He reports a roughly 30% inference speed improvement on an Apple-silicon experiment by fusing unnecessary separate kernels and reducing launch overhead.
- Further gains may come from hardware interconnection, orchestration, and hardware-aware agent harnesses, making local performance a full-stack problem.
local-aiapple-siliconinference-optimization
Original description
Running GLM 5.1, a trillion parameter model released the day before this workshop, across four Mac Studios costs around $40,000 in hardware and tops out at roughly 20 tokens per second. Alex Cheema from EXO Labs thinks both numbers have about 100x left in them. The workshop covers what that 100x looks like across the stack: kernel fusion that recovered 30% performance on Qwen 3.5 from inefficiencies nobody had noticed, RDMA integration that cut node to node latency from 300 microseconds to single digits and made tensor parallelism actually scale, and the case for splitting prefill onto compute dense hardware and decode onto high bandwidth hardware. The live demo runs GLM 5.1 across four Mac Studios connected by Thunderbolt 5 and cuts large prompt inference roughly in half by offloading prefill to an RTX Spark. Speaker info: / alex-cheema https://github.com/alexcheema Timestamps: 0:00 Introduction to EXO Labs and the mission to democratize frontier AI 2:00 The current state of AI: centralized cloud systems vs. the need for local infrastructure 7:40 Technical challenges: kernel efficiency and the overhead of separate kernel launches 9:50 The importance of the software harness in optimizing inference performance 10:35 Understanding inference constraints: compute-bound vs. memory-bound operations 11:28 The distinction between prefill and decode phases in LLM inference 13:07 Requirements for efficient local decoding: memory capacity, bandwidth, and energy efficiency 15:50 The concept of 'Intelligence per Joule' as a performance metric 16:45 Advancements in consumer hardware: higher memory capacity and bandwidth on Apple Silicon 18:50 Q&A: The future of consumer appetite for local inference hardware 20:17 Discussing the cost and performance of running trillion-parameter models like GLM 5.1 22:05 The 100x potential: How code design across the stack improves performance 26:35 Future outlook: Bifurcation of local vs. cloud use cases and diminishing returns of model size 38:55 Heterogeneous hardware strategies: Combining compute-dense and bandwidth-heavy devices 41:47 Demo: Using an Nvidia RTX Spark to accelerate prefill on a Mac cluster 48:40 Software architecture: Automating cluster orchestration with EXO 53:00 Challenging the 'batching' necessity: Multi-agent systems, search, and continual learning 1:05:22 Rethinking cloud economics and renting use cases instead of hardware 1:20:55 Demo technicalities: Event sourcing and cluster node discovery 1:32:20 Closing thoughts: Transparency in benchmarks and the future of open-source model evaluation