← back
Stop Renting Your Cognitive Infrastructure - Thiyagarajan Maruthavanan, Kalmantic Labs
Takeaway
Evaluate inference ownership using workload economics, operational control, and reproducibility requirements as applications mature.
Summary
- The speaker describes unexpectedly large inference bills for a music-analysis app and an API-key theft that caused further spending.
- Moving workloads to a local DGX system exposed memory constraints, while enterprise users raised concerns about rate-limit control, vendor dependencies, and reproducibility.
- The talk distinguishes rented APIs, hosted or local token factories, and owned infrastructure, arguing that established workloads deserve a fresh ownership assessment.
- Context compression and waste reduction remain useful across deployment models; the speaker presents Just Token Max as an open-source optimization project.
inference-economicsself-hostingtoken-costs
Original description
I pointed my lab at one problem, inference, after 200 users burned $1,000 in credits and the math just wouldn't close. So I built the thing, felt the cost, and went looking for why renting intelligence never pencils out. Turns out everyone in this market sells a gospel shaped like their own invoice. Jensen: build a token factory. Nadella: don't even think about the meter. Fireworks: own your model (on our infra). Three smart people, three different layers, three pitches that all end at "keep paying us." My rule: rent to learn, own to run. Rent the model while you're hunting PMF, own the inference for the part you'd have to answer for. I moved my own agents off the Anthropic API onto owned infra, open-sourced the piece that stops the bleed, and got few things badly wrong on the way Speakers: Thiyagarajan Maruthavanan (Kalmantic Labs): Thiyagarajan M (Rajan) runs an agentic lab focused on AI inference and agent harness, has built open source tools and other products to shape work on it, and authored a book on peak inference performance. X/Twitter: https://x.com/mtraja LinkedIn: / thiyagarajan GitHub: https://github.com/mtr7x