0.3 How LLMs work · FDE Companion

The model is a function: text in, text out

Strip away the chat window and an LLM is one thing: you send it text, it predicts what text comes next, one piece at a time. That's it. As Book §3.1 puts it, the model "doesn't read files. It doesn't run commands. It doesn't remember the last session." Everything else you've seen AI do is software wrapped around that function.

Five ideas explain most of what you'll argue about in the bootcamp.

Instructions, the conversation so far and any documents go into the context window as tokens. The model reads the whole window and predicts the next token, which is added to the answer, one token at a time until it stops. — One call. Everything the model sees has to fit in the context window, and the answer comes out one token at a time.

1. Tokens

Models don't read letters or words. They read tokens, chunks of text roughly ¾ of an English word on average. "Unbelievable" might be three tokens; a line of code or a non-English sentence uses more. Prices, speed limits and context limits are all measured in tokens. Try the token counter under this reading.

2. The context window

The context window is the maximum number of tokens the model can consider in one call: your instructions, the conversation so far, any documents you included, and its answer. Anything outside the window doesn't exist for the model. Bigger windows help, but filling them is slow and expensive, which is why retrieval (RAG) exists: send the 5 relevant pages, not the 5,000.

3. Statelessness

Each API call is independent. The model does not remember your previous call. When a chatbot "remembers" you, the app is resending the whole conversation every turn. That has two consequences you'll design around constantly:

Long conversations get more expensive per turn, because the history is resent.
Memory is a product decision: what to keep, what to summarise, what to drop. Claude Code's CLAUDE.md is exactly this, a file the harness puts back into context every session.

4. Temperature and non-determinism

At each step the model picks the next token from a probability distribution. Temperature controls how adventurous that pick is: low for extraction, classification and code; higher for brainstorming. Even at low temperature, outputs vary, which is why the bootcamp spends real time on evals.

5. Cost, and open vs. closed

You pay per input token and per output token (output usually costs more). A small model can be 50× cheaper than a frontier one. The usual move is to route easy work to small models and hard work to big ones. The cost calculator under this reading shows how fast it adds up.

Closed models (Claude, GPT, Gemini) are accessed via API: strongest quality, zero ops, but data leaves your infrastructure. Open-weight models (Llama, Qwen, Mistral) can run on your own servers. Enterprises choose them for data residency, cost at scale and control. In the bootcamp you'll do both: call hosted models through one API, then serve a model yourself on vLLM.

Anatomy of an API call

Every chat-style API looks roughly like this:

POST https://<model provider>/v1/chat/completions
{
  "model": "a-model-name",
  "messages": [
    { "role": "system", "content": "You are a concise assistant." },
    { "role": "user",   "content": "Explain tokens in one sentence." }
  ],
  "temperature": 0.2
}

The response includes the reply and a usage block telling you exactly how many tokens you were charged for. Notice there's no "conversation id": if you want the model to see earlier messages, you put them in messages yourself. Claude Code sends a call shaped like this on every turn of every session. In the bootcamp you'll write them yourself.

Try it

Hands-on, in your browser.

Token counter · approximate

Forward deployed engineers embed with a client, find the workflow that hurts, and ship an agent that fixes it. Tokens are what the model actually reads.

35tokens

26words

152characters

1.35tokens per word

Try pasting code, a long German word, or an email address and watch the token count jump. The model never sees letters, only these pieces, which is why LLMs are bad at counting characters.

The model forgets · two API calls

Call 1

Hi, I'm Sara. I run ops at a logistics company.

Nice to meet you, Sara! How can I help?

Call 2 · what actually gets sent

{
  "messages": [
    {
      "role": "user",
      "content": "What's my name and what do I do?"
    }
  ]
}

I don't know your name. I don't have any information about you.

The app resends the conversation history with call 2

The model has no memory between calls. "Memory" is always the application putting earlier text back into the context, and you pay for those tokens again every time.

Cost calculator · illustrative prices

Model tier

Calls per day

Input tokens per callSystem prompt + history + retrieved documents + the question

Output tokens per call

$0.02per call

$39.00per day

$1,170per month

Try doubling the input tokens (stuffing a whole document into every call) and compare that with switching model tiers. Most real cost wins come from sending less context, not from haggling over price.