← back
Under 5 minutes to a deployed LLM endpoint — Audry Hsu, RunPod
Takeaway
Preconfigured serverless GPU endpoints simplify deployment, while worker settings determine scaling and cold-start behavior.
Summary
- RunPod offers GPU-backed pods, autoscaling serverless endpoints, training clusters, and a hub of deployable repositories.
- The console demo deploys a Hugging Face LLM through a preconfigured vLLM repository and adjusts its context length.
- GPU choices, maximum workers, and always-on workers expose cost, capacity, and cold-start tradeoffs.
- The first demonstrated request waits about 41 seconds in the queue during initialization and executes in roughly 1.5 seconds; telemetry separates delay from execution.
runpodserverless-gpuvllm
Original description
Two failed crypto mining rigs in a basement in 2022. The founders posted on Reddit offering the GPUs for free in exchange for feedback. That is the origin of RunPod, now at $120 million in annual recurring revenue with 500,000 developers on the platform. The demo runs in under five minutes: pick a model from the Hub, configure a context window, deploy a serverless endpoint on H100s. First request queues for 41 seconds on cold start while the container initializes and the model downloads. Every request after that executes in about 1.5 seconds. You pay only while a worker is handling a request. Speaker info: / audry-hsu