🏗️ AI Infrastructure
GPU clusters, training stacks, autoscaling inference, data pipelines, feature stores, observability for AI workloads.
The workflow
flowchart LR
A[Workload] --> B{Stage}
B -->|Train| C[Multi-GPU cluster<br/>NVLink / RDMA]
B -->|Serve| D[Autoscaling fleet<br/>spot + reserved]
B -->|Data| E[Lakehouse +<br/>feature store]
C --> F[Observability<br/>traces + metrics]
D --> F
E --> F
F --> G[Cost / perf<br/>tuning loop]
Training is bursty; serving is steady; the bill is paid by serving.
Key takeaways
Videos (83)
From fork() to Fleet: Designing an Agent Sandbox Cloud — Abhishek Bhardwaj, OpenAI
Design agent sandbox clouds around isolation, persistent state, and the distinct performance needs of training and products.
Thinner Agents on a Smarter Substrate: The Ontology-based Semantic Layer — Emil Eifrem, Neo4j
Move shared business-to-data mappings and execution knowledge into a governed semantic layer so agents can remain simpler.
The Production AI Playbook: Deploying Agents at Enterprise Scale — Sandipan Bhaumik, Databricks
Define measurable success, traceable behavior, reliable data, and operational ownership before scaling enterprise agents.
The Geopolitics of AI Infrastructure - Dylan Patel, SemiAnalysis
Export controls have not stopped Chinese compute scale-up, and Middle East gigawatt-scale builds are reshaping where frontier training happens.
AI-Native Organisations Run on Skills: How to Structure and Scale Them — Imad Touil, QuantumBlack
Scale agent adoption by managing skills as shared software assets that encode the organization's real workflows.
Why Your Enterprise Tech Stack Isn’t Ready for AI Agents — Christopher Lovejoy & Saul Howard
Design production agents around auditable state and enterprise controls from the start instead of adding them after a promising demo.
Agentic SDLC at Uber — Uday Kiran Medisetty & Adam Huda, Uber
Scaling agentic software delivery requires shared gateways, ready execution environments, governed skills, and centralized context.
CI/CD Is Dead, Agents Need Continuous Compute and Computers — Hugo Santos and Madison Faulkner
Agent-driven development needs a continuous-compute substrate that replaces PR-centric CI/CD with high-throughput, machine-paced merging.
The Rise of CaaS: Context-as-a-Service for Agentic AI — Omer Primor, Bright Data
Useful web context combines fresh discovery with structured entities and historical data tailored to a domain.
How to Build Your Own AI Data Center in 2025 — Paul Gilbert, Arista Networks
Enterprise AI data centers need rail-optimized, isolated backend GPU networks tuned for job-completion time, sized differently for training vs inference.
WTF Is the Context Layer? The Missing Infrastructure for Production Agents — Prukalpa Sankar
Treat shared business knowledge, expertise, and update propagation as infrastructure for a coordinated team of agents.
Anthropic's Applied AI team on the Evolution of Agentic Surfaces
Agent infrastructure must evolve with model capabilities, and obsolete workarounds should be removed as models improve.
Efficient Reinforcement Learning – Rhythm Garg & Linden Li, Applied Compute
Applied Compute scales enterprise RL by trading off policy staleness against throughput in asynchronous pipeline RL — the efficient frontier between speed and learning stability.
Why Eval++ Is the Next Great Compute Primitive — Sunil Pai & Matt Carey, Cloudflare
Persistent agent coordination and lightweight isolated code execution form complementary building blocks for agent infrastructure.
Arrakis: How To Build An AI Sandbox From Scratch - Abhishek Bhardwaj, OpenAI
Run AI-agent-generated code inside lightweight microVMs with snapshot/restore rather than containers — speed and isolation both matter at agent scale.
AI Platform Engineering: Patrick Debois
Build a centralized AI platform (models, vectors, connectors, observability, eval monitoring) so feature teams don't each reinvent rag-ops and governance.
Rishabh Garg, Tesla Optimus — Challenges in High Performance Robotics Systems
High-performance robotics depends on pipelined, synchronized comms threads — and most 'bad policy' bugs are actually CAN/thread timing bugs you can only see with bus-level logging.
Productionizing GenAI Models – Lessons from the world's best AI teams: Lukas Biewald
In GenAI, the learnings (not the code) are your IP — track every experiment automatically so iteration time, not feature velocity, becomes your competitive edge.
The Missing Primitive for Agent Swarms — Lou Bichard, Ona
Scaling coding agents requires explicit coordination and dependable execution environments in addition to the agents themselves.
Can LLMs Write Fast Multi-GPU Kernels? — Simran Arora, Together AI
Fast multi-GPU code requires reasoning about communication topology and data movement, which need dedicated evaluation beyond single-device kernel performance.
We accidentally made an AI platform: Jamie Turner
A reactive backend platform turned out to be the right substrate for shipping AI apps with confidence, with vector indexes and component libraries as natural extensions.
Agents Are Where Microservices Were in 2015 — Roberto Milev & Uday Kanagala, Navan
Production agents need explicit runtime, memory, context, and observability layers.
Two Roads to Durable Agents: Replay vs. Snapshot — Eric Allam, CEO, Trigger.dev
Durable agents need either replay-style journaling (Temporal) or snapshot-style state capture; replay's determinism constraints make it awkward for LLM-driven workflows.
Unlocking Developer Productivity across CPU and GPU with MAX: Chris Lattner
Framework fragmentation is the bottleneck for production GenAI; MAX targets one runtime spanning CPU and GPU to bridge research and prod.
AI Agents Are Just Distributed Systems Now — Salman Munaf, TikTok
Apply distributed-systems controls around agent decisions, especially where actions change external state.
Building LinkedIn's GenAI Platform — Xiaofeng Wang
LinkedIn's GenAI platform evolved through three generations to support multi-agent systems with distributed orchestration, a skill registry, layered memory, and OpenTelemetry-based observability.
The Half Life of Agent Infrastructure — Ben Kus, Box
Design enterprise agent infrastructure to accommodate rapid replacement of models and orchestration patterns.
Road to 5 Million Tokens: Breaking Barriers in Long Context Training — Max Ryabinin, Together AI
Reusing and offloading activation memory can extend context-parallel training beyond limits reached by standard techniques.
Your agent architecture has a half-life of 6 months — Dan Farrelly, CTO, Inngest
Decouple durable execution from fast-changing models and ephemeral compute to reduce architecture rewrites.
Lobster Trap: OpenClaw in Containers from Local to K8s and Back — Sally Ann O'Malley, Red Hat
Container packaging gives agent deployments familiar tools for isolation, portability, secret handling, and state recovery.
AI Kernel Generation: What's working, what's not, what's next – Natalie Serrino, Gimlet Labs
AI agents can already deliver double-digit kernel speedups on real workloads by iterating compile-run-profile loops, but struggle on the most complex kernels — promising for cross-hardware porting.
x402 isn’t good (yet) — Jan Curn, Apify
Agent payments unlock paid tools, but x402 adoption still requires realistic expectations about an emerging protocol.
Gadgets: Personal app vibe coding that is actually safe — Kenton Varda, Cloudflare
Personal software needs independently customizable app instances and reusable code templates.
Agent Spending Without Controls — Rodrigo Coelho & Pranav Maheshwari, Edge & Node
Enterprise agent payments need a financial control layer alongside access to paid tools.
Productionizing LLM Gateways: Architecture, Tradeoffs and Hard Lessons — Kanish Manuja, Twilio
Production gateways need configurable failure policies and per-route observability rather than generic retries.
The Pipeline Is Dead - Iris ten Teije, Sky Valley Ambient Computing
Adaptive software requires architectural boundaries that make individual customization isolated and reversible.
GPU-less, Trust-less, Limit-less: Reimagining the Confidential AI Cloud - Mike Bursell
TEE-based confidential AI plus a decentralized marketplace (Super Protocol) enables training and inference on sensitive data without trusting any cloud provider.
When AI Agents Pay and Sellers Monetize: Building x402 Apps on AWS — Anil Nadiminti, AWS
Machine-readable payment flows can connect agent access to content monetization with less manual setup.
Why Your AI Agent Needs a Wallet: USDC and Nanopayments — Harshal Bhangale, Circle
A wallet with enforced spending limits can let an agent independently use paid services needed for a task.
Why, and how you need to sandbox AI-Generated Code? — Harshil Agrawal, Cloudflare
Always sandbox AI-generated code with capability-based security — V8 isolates for fast lightweight execution, containers when you need a full Linux environment.
What Breaks When You Build AI Under Sovereignty Constraints - Bilge Yücel, deepset GmbH
Design sovereignty into data flows and component interfaces early, because retrofitting it exposes hidden dependencies throughout the stack.
Scaling the Next Paradigm of Heterogeneous Intelligence — Adrian Bertagnoli, Callosum
Match each subproblem to suitable models and hardware to improve the cost-performance frontier of agent systems.
Production software keeps breaking and it will only get worse — Anish Agarwal, Traversal.ai
Autonomous incident debugging requires fusing causal ML, semantics and custom agent control flow — neither AIOps, plain LLMs nor ReAct agents alone can do it.
Deterministic Infra for Non-Deterministic AI Agents - Nishant Gupta, Meta Superintelligence Labs
Keep probabilistic decision-making inside deterministic infrastructure that bounds failures, cost, and authority.
Compilers in the Age of LLMs — Yusuf Olokoba, Muna
A Python compiler with LLM-assisted verification can turn AI inference code into portable native binaries that run anywhere, sidestepping container-based deployment.
Why We Don't Need More Data Centers - Dr. Jasper Zhang, Hyperbolic
Distributed GPU marketplaces are a faster, cheaper way to satisfy AI compute demand than waiting 7+ years for new hyperscale data centers.
The Agent Behind the Curtain: Building the Oz Cloud Agent Platform — Safia Abdalla, Warp
A cloud-agent platform should absorb infrastructure complexity while exposing composable primitives and preserving developer choice.
Scaling AI Agents Without Breaking Reliability — Preeti Somal, Temporal
Temporal's decade-old durable workflow engine maps cleanly onto agent reliability needs and is now seeing Python SDK overtake others as agents go to production.
AX is the only Experience that Matters - Ivan Burazin, Daytona
Build tools for agents (Agent Experience) not for humans-with-AI — speed, API-first, agent-readable docs, and autonomy-by-default are the bar.
FinOps for AI Agents: Who Spent All the Tokens? — Tisha Chawla & Susheem Koul, Microsoft
Govern cumulative agent-run spending with attribution and active runtime controls.
The Hierarchy of Needs for Training Dataset Development: Chang She and Noah Shpak
Training-data infra is the bottleneck; Lance format + materialization service enables Character.AI's iteration speed by satisfying filter+shuffle+blob-stream simultaneously.
Platforms for Humans and Machines: Engineering for the Age of Agents — Juan Herreros Elorza
To make coding agents productive in enterprise, redesign internal platforms around self-service APIs with shift-left feedback so agents (and humans) can iterate without humans-in-the-loop.
I Run a Fleet of AI Agents Across Three Machines. Here's What Broke. - Kyle Jaejun Lee, KRAFTON
Durable state, isolated environments, and centralized review become essential as agent fleets spread across machines.
How Web Data Infrastructure Powers the Next Generation of AI — Patricija Žemaitytė, Oxylabs
AI data infrastructure must balance freshness, scope, latency, and reliability under actual collection conditions.
Building safe Payment Infrastructure for the autonomous economy — Steve Kaliski, Stripe
Keep payment authority in deterministic, auditable infrastructure with narrowly scoped credentials and enforced spending limits.
OpenLLMetry is all you need
OpenLLMetry brings the OpenTelemetry standard and ecosystem to LLM apps so you get vendor-neutral, drop-in observability across providers, vector DBs and agent frameworks.
Emulated: The Data for Fully Autonomous Software Engineers and Companies — Joseph Wang
Training autonomous engineers requires environments that capture ongoing operational responsibility beyond producing a patch.
Infrastructure for the Singularity — Jesse Han, Morph
Future agentic infra needs sub-second VM snapshot/branch/replicate primitives so agents can fork environments faster than humans can deploy them.
Taking Reinforcement Learning Cross Datacenter — Nan Jiang, Modal
Separating rollout capacity from the training cluster becomes practical when policy synchronization transmits compact exact changes.
GPU Cloud Deployment Without Leaving Your IDE — Audry Hsu, RunPod
Function-level cloud GPU execution can make model development feel like ordinary local iteration.
GTM Engineering: The Technical Bits — Everett Berry, Clay
Fast GTM execution depends on trustworthy, continuously maintained data and robust orchestration across fragmented tools.
Your Agent Didn't Fail. Your Harness Did. — Vinoth Govindarajan, OpenAI
Production agent reliability depends on durable state ownership, ordered mutations, and verifiable action receipts in the harness.
Stop Renting Your Cognitive Infrastructure - Thiyagarajan Maruthavanan, Kalmantic Labs
Evaluate inference ownership using workload economics, operational control, and reproducibility requirements as applications mature.
The Missing Layer in Agentic AI — Giedrius Šteimantas, Oxylabs
Web agents benefit from a validated, lightweight data-access layer that reserves browser interaction for tasks that need it.
Your Agents Need a Save Button - Hamza Tahir, ZenML
Checkpointing executable agent state turns production observations into repeatable experiments on models, tools, and policies.
Context Platform Engineering to Reduce Token Anxiety — Val Bercovici, WEKA
Treat agent context as a tiered storage problem and maximize KV-cache hit rate at the platform layer — that beats prompt-cache arbitrage and most other inference optimizations.
Keynote: The AI developer experience doesn't have to suck – why and how we built Modal
Sub-second container start + Python-native serverless makes Modal feel like local iteration while scaling to thousands of GPUs.
Serving 2 Million Models Without Melting: Scaling the Hugging Face Hub — Arek Borucki, Hugging Face
Separate storage responsibilities and optimize read paths while reserving the primary database for work that requires it.
Infra behind Krea 2: How to train and serve at scale — Gabriel Jorge Menezes, Krea.ai
Large-scale training reliability depends on detailed hardware and network metrics plus fast, frequent recovery checkpoints.
Conquering Agent Chaos — Rick Blalock, Agentuity
Agents need agent-specific infra — long runtime, stateful routing, framework-agnostic deployment — not stateless serverless.
Learned Execution Graphs for Anomaly Detection & Drift in APIs — Ritvik Pandya, JP Morgan Chase
Execution graphs make API anomalies actionable by locating deviations and distinguishing failures from legitimate workload drift.
Your LLM Stack Is a 2008 Database With Better Marketing — Lovina Dmello, NVIDIA
Production AI security depends on correctly configured infrastructure and practical defense costs as much as model-specific protections.
Infra that fixes itself, thanks to coding agents — Mahmoud Abdelwahab, Railway
Combine durable workflows with a headless coding agent (OpenCode) so infrastructure issues become reviewable pull requests instead of pages.
Research to Reality: Bringing Frontier ML Research to Production - Vaidas Razgaitis, Higharc
A clear research handoff document and repeatable service architecture help frontier ML prototypes become maintainable production features.
Accelerating Mixture of Experts Training With Rail Optimized InfiniBand Networking in Crusoe Cloud
Rail-optimized InfiniBand on green-powered Crusoe Cloud cuts the all-reduce communication penalty that otherwise idles GPUs for 25-30% of MoE training time.
Luminal - Search-Based Deep Learning Compilers - Joe Fioti
Reduce deep learning to ~12 primitives and let search-based compilers generate the fast code — a path to vastly simpler ML stacks that still hit peak hardware performance.
Privacy-Preserving Intelligence — Steve Korshakov, Bee (acq. Amazon)
Always-on personal agents can combine cloud execution with strong privacy controls by tying key access to attested, auditable workloads and limiting key lifetime.
How agents broke app-level infrastructure - Evan Boyle
Agentic workloads break Web 2.0 infrastructure assumptions about latency and reliability; we need durable execution layers built for seconds-to-hours requests.
Building Agentic Applications w/ Heroku Managed Inference and Agents — Julián Duque & Anush Dsouza
Heroku now ships managed inference + MCP-based agents + pgvector so apps can attach AI and tools the same way they attach Postgres.
Substrate Launch: the API for modular AI
Substrate runs multi-model computation graphs as a coordinated cluster, replacing many slow API calls with microsecond inter-node hops and reliable structured outputs.
Continuous Profiling for GPUs — Matthias Loibl, Polar Signals
Always-on sampled profiling with eBPF + NVML + GPU time attribution finally gives the GPU equivalent of CPU flame charts in production.
Building agent fleet architectures your CISO doesn't hate — Lou Bichard, Gitpod
For regulated buyers, the right agent-fleet architecture is a substrate: customer owns the workload + source code on their cloud, vendor manages the control plane via minimal telemetry — not pure SaaS or pure self-hosted.
Accelerate your AI journey with Azure AI model catalog: Sharmila Chokalingam
Azure positions itself as a unified catalog + serving layer letting enterprises prototype, optimize, and operationalize across 1,600+ models with consistent APIs and data privacy.