The Infrastructure Layer
In part 3, we looked at the model. This is the machinery that trains it, serves it, and keeps it alive.
Most application engineers meet infrastructure through an API. That abstraction is useful until latency spikes, capacity disappears, or the bill stops making sense. Then the details matter.
Compute and Memory
Model workloads are dominated by tensor operations and memory movement. GPUs and other accelerators perform many operations in parallel, while CPUs handle request management, preprocessing, databases, and control-plane work.
For inference, model weights and runtime state must fit within available memory. The rough weight footprint is easy to estimate:
parameter count × bytes per parameter = weight memory
The real requirement is larger. You also need space for activations, the KV cache, framework overhead, and concurrent requests. Quantization can reduce the weight footprint, but it changes the quality and performance profile.
Storage
Object storage holds datasets, checkpoints, model artifacts, and logs. A model registry adds versioning, metadata, lineage, and approval state.
The serving path usually needs a warm copy of the model in fast memory. If every replica repeatedly downloads large weights or if the storage path is slow, expensive accelerators sit idle while the system waits.
Storage design is therefore part of serving design. Decide what is durable, what is cached, how artifacts are promoted, and how a bad version is rolled back.
Networking
There are two different networking problems:
- getting user requests to a healthy model replica,
- moving data quickly between machines when a model or workload is distributed.
API gateways handle identity, rate limits, validation, and routing. Load balancers distribute traffic. Internal links and accelerator interconnects determine how efficiently a large model can share work across devices.
Streaming adds another constraint: the first token and the tokens after it have different latency characteristics. Measure both instead of hiding everything behind one average response time.
Serving and Orchestration
An inference server manages model loading, batching, scheduling, streaming, and sometimes parallelism across devices. Systems such as vLLM provide optimized serving primitives; other stacks make different trade-offs.
Orchestration schedules workloads, exposes accelerators, restarts unhealthy processes, and scales capacity. Kubernetes can schedule GPUs through vendor device plugins and expose them as schedulable resources. It is a control plane, not a magic solution: you still need sensible requests, health checks, capacity planning, and rollout policies.
Reliability and Safety
Production infrastructure needs more than uptime metrics:
- request rate, queue depth, and error rate,
- time to first token and time between tokens,
- accelerator utilization and memory pressure,
- model version and configuration,
- token usage and cost,
- trace context across gateway, application, and model server.
Security follows the same principle as the application layer: least privilege. Separate development and production, scope service credentials, protect model artifacts, isolate workloads, and make destructive actions difficult to perform accidentally.
Deployment Choices
You can use a hosted model API, a managed inference service, your own cloud cluster, on-premises hardware, or an edge device. The choice depends on data handling, latency, traffic shape, reliability requirements, and operational capacity.
Self-hosting is not automatically cheaper. Hardware utilization, engineering time, upgrades, redundancy, and incident response are part of the cost. Hosted infrastructure is not automatically safer either. Understand the provider’s isolation, retention, region, and failure guarantees.
Where the Bottlenecks Hide
Infrastructure failures usually come from a mismatch between one layer and another:
- the model does not fit in available memory,
- storage cannot load artifacts fast enough,
- batching improves throughput but breaks latency targets,
- autoscaling reacts after the queue is already long,
- a rollout changes model behavior without a quality gate,
- observability records CPU while ignoring tokens and model versions.
The fix is measurement. Profile the full request path, then optimize the constraint that actually limits the system.
Summary
Infrastructure turns a model artifact into a service. Compute performs the work. Memory and storage hold the state. Networking moves requests and data. Serving and orchestration manage execution. Observability and security make the system operable.
The infrastructure layer is where AI meets physics: finite memory, finite bandwidth, finite budgets, and imperfect machines. Good infrastructure does not remove those limits. It makes them visible and spends them deliberately.