The AI Stack Explained
Everyone is building with AI. Far fewer people understand what they are building on.
That matters the first time something breaks. A model can return nonsense, an inference service can run out of memory, or a data pipeline can quietly poison every answer above it. You fix those failures faster when you can see the whole system.
This series is a map of that system.
Three Layers
The AI stack is a useful abstraction, not a law of nature. It groups the system into three layers.
The application calls the model. The model runs on infrastructure. Data, evaluation, and feedback move across all three.
The boundaries are not perfectly clean. Retrieval belongs to the application in one system and the platform in another. A model provider may own the infrastructure while your team owns the prompt, routing, and evaluation. The layers are still useful because they give you a place to start when diagnosing a problem.
The Application Layer
This is the part users see: chat interfaces, copilots, recommendation systems, document workflows, and agents that take actions.
The application turns an ambiguous human request into a structured task. It manages authentication, business rules, prompt construction, tool access, output handling, and the user experience around uncertainty.
The model is not the product. The application decides when to call it, what context to provide, what actions it may take, and what happens when the answer is wrong or incomplete.
Read part 2: The Application Layer
The Model Layer
The model is the learned component: an architecture plus weights produced by training. During inference, the model turns input tokens—or other modalities such as images or audio—into an output.
Models differ in capability, latency, context length, modality, cost, and deployment requirements. A model can be accessed through an API, downloaded and served yourself, adapted with fine-tuning, or paired with retrieval when its built-in knowledge is not enough.
The Infrastructure Layer
Infrastructure makes training and inference possible. It includes accelerators, memory, object storage, networking, schedulers, serving systems, observability, and deployment controls.
The infrastructure layer determines whether a model fits in memory, how quickly it responds, how many requests it can handle, and what each request costs. It is where a promising prototype meets physics and a budget.
Read part 4: The Infrastructure Layer
Follow One Request
Consider a user asking a support assistant a question.
The application authenticates the user, retrieves relevant documents, builds the model input, and decides whether the model may call a tool. The model generates an answer or a tool request. The serving system schedules that work on available hardware. The infrastructure returns tokens and records latency, errors, and resource usage. The application applies policy and streams the answer back.
The user experiences one interaction. Three layers made it happen.
Use the Stack to Debug
When an AI feature fails, start by locating the layer:
- Wrong answer: inspect context, retrieval, model behavior, and evaluation.
- Slow answer: inspect queueing, batching, model size, network hops, and token generation.
- High cost: inspect routing, output length, cache use, and accelerator utilization.
- Unsafe action: inspect tool permissions, application policy, confirmation steps, and audit logs.
- Frequent outages: inspect capacity, deployment strategy, health checks, and recovery paths.
The point of a stack is not to memorize product names. It is to understand dependencies. A change below you changes the constraints above you.
The Mental Model
The application gives the model a job. The model supplies learned capability. Infrastructure supplies the resources and controls that let the system run.
Good AI engineering means reasoning across all three. You do not need to be a distributed-systems expert to build an application, but you do need to know when your application problem is really a model or infrastructure problem.
That is the map. The rest of the series fills in the terrain.