The Model Layer

In part 2, we looked at the application around a model. Now we look at the model itself.

A model is not a database of answers and not a person trapped in a server. It is a parameterized function: an architecture plus learned weights that transforms an input into an output.

From Tokens to Output

TRAIN

data → weights

learn patterns

SERVE

tokens → output

run inference

ADAPT

retrieval · tools

add runtime knowledge

Training changes the weights. Runtime systems change the input.

For a language model, text is split into tokens. The model processes those tokens through layers of learned transformations, then produces a probability distribution over what might come next.

Generation is usually autoregressive:

input tokens → predict next token → append token → repeat

The model does not retrieve a sentence from a filing cabinet. It repeatedly estimates what follows from the context it was given.

Training

Pretraining exposes a model to a large corpus and adjusts its weights to reduce prediction error. The result is a general capability, not a guarantee that every answer is true.

Later stages shape how the model behaves. Instruction tuning teaches it to respond to tasks. Preference or reinforcement methods push it toward outputs people or evaluators prefer. Evaluation tests whether those improvements survive outside the training examples.

The important distinction is between knowledge in the weights and information supplied at runtime. If the answer depends on private, current, or exact facts, use retrieval or a tool instead of assuming the model’s weights are enough.

Inference

Inference is the production path: load the weights, tokenize the input, run the forward pass, and generate output. It is constrained by memory, bandwidth, latency, concurrency, and cost.

The context window is the amount of input the model can process in one call. A larger window helps with long documents, but it does not remove the need to select relevant context. More tokens also mean more work and more cost.

The KV cache stores intermediate attention data for tokens already processed. Reusing it avoids repeating some computation while generating the rest of a response. Batching lets a server share hardware across requests, but it creates a trade-off between throughput and latency.

Quantization stores weights with fewer bits. It can reduce memory use and improve serving efficiency, with a quality trade-off that must be measured on the task that matters.

Adapting a Model

When a model is not good enough, change the right layer:

Prompting: clarify instructions and output requirements.
Retrieval: supply relevant external knowledge at runtime.
Tool use: let the model query systems or perform bounded actions.
Fine-tuning: change behavior using task-specific examples.
Model replacement: choose a model with better capability for the task.

Fine-tuning is not a substitute for current data. Retrieval is not a substitute for reasoning. A bigger model is not a substitute for verification.

Parameter-efficient methods such as LoRA update a smaller set of parameters or add trainable adapters instead of changing every weight. They reduce training cost, but they still require representative data and a real evaluation set.

Not Every Model Does the Same Job

The model layer includes more than chat models:

language models generate and transform text,
embedding models map content into vectors for similarity search,
vision-language models combine image and text inputs,
speech models transcribe or synthesize audio,
rerankers score retrieved candidates,
classifiers and moderation models make narrower decisions.

Choose by task, not by reputation. Measure quality, latency, cost, privacy, and operational fit on your own workload.

Hosted or Self-Hosted?

Hosted APIs are usually the fastest path to a product. You trade control for a managed service and usage-based cost.

Open-weight models give you more control over deployment, data handling, and adaptation. You also inherit serving, capacity, updates, and security work. The break-even point depends on traffic, hardware, engineering time, and the cost of being wrong.

The right comparison is not “best model.” It is “best model-system combination for this task.”

Model Quality Is Not the Same as Product Quality

A model benchmark measures a narrow capability under a defined test. A product has users, tools, permissions, latency budgets, and failure modes.

The application decides what context reaches the model. Infrastructure decides whether the model can serve that context in time. Evaluation decides whether an output is good enough. Model quality is important, but it is only one part of the system.

Summary

Models provide learned capability. Inference turns that capability into outputs. Adaptation, retrieval, tools, and evaluation determine how useful those outputs become.

Treat model choice as an engineering decision, not a leaderboard decision.