Running a large language model on hardware you control is no longer exotic, but the gap between a model that loads and a service that holds up under load is where most self-hosting efforts stall. This book closes that gap by building the thing, one working system that grows across all fifteen chapters.

You begin with an 8B model on a single card, the kind of hardware you can rent for pennies an hour or already own. From there the book adds what real traffic demands: an inference engine chosen on its merits, a control plane that provisions GPUs and tears them down again, a router and queue that decide what runs next, and cache-aware routing that stops a conversation being re-read from scratch on every turn. Part III moves past what one card can hold, placing a 70B deployment across a pool of GPUs selected on the network between them rather than the sum of their memory, and shopping for that capacity across several providers at once.

The final part goes to the frontier. You will serve a 753-billion-parameter sparse model on eight rented H200s, then measure what it costs per session as concurrency climbs, and find the point where a larger batch stops making each session cheaper. That result explains how commercial inference providers price what they sell, and it tells you when self-hosting wins and when it does not.

Costs are measured rather than estimated throughout. Every figure in the final part is regenerated from a committed data file, and the book says plainly which numbers are measurements and which are predictions. The complete control plane is available as an open-source companion repository, written in Go, so you can run what you read.

You Will:

• Build a working inference control plane in Go that provisions GPUs across providers, deploys engines onto them, routes and queues requests, and reports what it is doing.
• Learn how inference actually consumes a GPU, covering KV cache growth, quantization formats and their real cost, tensor and expert parallelism, and why a fast interconnect matters more than total memory.
• Measure the economics from real runs, working out per-session cost as batch size and context length grow, and identifying the point where self-hosting stops beating a commercial API.

This book is for:

Backend, systems and platform engineers who want to run LLMs on infrastructure they control. Also useful to infrastructure and DevOps engineers building internal AI platforms, and to technical founders with data-residency or cost constraints that rule out third-party APIs. No machine learning background is assumed. Readers should be comfortable with a systems language such as Go or Python, with containers, and with HTTP APIs.

Sriram Panyam has spent over 20 years building large-scale distributed systems, cloud and AI platforms and the data analytics that run on them, at Google, LinkedIn, Amazon, several startups and now General Motors. At Google Cloud he led engineering teams working on petabyte-scale workloads and control planes on Cloud Dataflow. At LinkedIn he built and managed identity and analytics platforms serving more than a billion members. He has mentored a number of senior technical leaders, and writes about systems engineering and AI infrastructure.

His recent work is on AI platform infrastructure and the standards beneath it, including the Model Context Protocol (MCP). Most of that effort goes into conformance, the test suites that decide whether an implementation matches the specification rather than merely looking like it does, and into the reference implementations that show what a correct one looks like.

He holds engineering degrees from the University of Melbourne.