Generate One Token at a Time - demystifying the LLMs

The fastest way to stop finding LLMs mysterious is to run one on hardware slow enough that you can watch it work. A Raspberry Pi is perfect for this - not because it is a good place to serve a model, but because autoregressive text generation stops being an abstraction when each token takes a visible fraction of a second to appear.

There is no cloud involved here, no API key, and nothing leaves the device.

What you need

• A Raspberry Pi 5. 8GB comfortably runs models up to about 3 billion parameters; 4GB will handle the 1B class fine.
• A 64-bit OS. Run uname -m - it must print aarch64. A 32-bit install will not work.
• Active cooling. The Pi will sit at 100% on all four cores for minutes at a time, and a throttled Pi halves your speed.
• Ideally an NVMe HAT or USB SSD. Model files are gigabytes, and loading them off a tired microSD card is painful.

Step 1: Prepare the Pi

sudo apt update && sudo apt full-upgrade -y
sudo apt install -y git cmake build-essential python3-pip python3-venv
uname -m          # expect: aarch64
free -h           # check how much RAM you actually have to play with

Step 2: The easy path - Ollama

Ollama wraps model download, quantisation choice and serving into one command. It is the shortest distance between a fresh Pi and a talking model.

# The official installer. To read it first rather than pipe to a shell:
#   curl -fsSL https://ollama.com/install.sh -o get.sh && less get.sh
curl -fsSL https://ollama.com/install.sh | sh

# Pull and run a small model - it will start streaming tokens at you
ollama run qwen2.5:0.5b

Pick the model to fit the RAM you have. These are rough figures for 4-bit quantised weights, and you want headroom on top for the OS and the context:

Step 3: The interesting path - llama.cpp

Ollama is convenient but it hides the machinery. llama.cpp is the C++ inference engine underneath most local-LLM tooling, and building it yourself exposes the parts worth understanding.

git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
cmake -B build
cmake --build build --config Release -j4     # ~10 minutes on a Pi 5

Then fetch a model in GGUF format - the single-file packaging that llama.cpp uses - and run it:

./build/bin/llama-cli -m ./models/qwen2.5-0.5b-instruct-q4_k_m.gguf \
  -p "Explain what a transformer is, in one paragraph." \
  -n 128 -t 4

That q4_k_m in the filename is the important bit. Quantisation is what makes any of this possible on a Pi: a model's weights are trained as 16-bit numbers, and quantisation stores them at roughly 4 bits instead. The model gets about four times smaller and measurably faster, at the cost of a little accuracy. Without it, a 3B model would need ~6GB of RAM just for weights; with it, ~2GB.

The -t 4 flag pins it to four threads - one per Pi core. More threads than cores makes it slower, not faster.

Step 4: The loop itself

Here is the thing the heading promises. Underneath the chat interface, a language model does exactly one job: given all the tokens so far, produce a probability for every possible next token. Everything else is a loop:

             "The Raspberry Pi is"
                        │
                        ▼  tokenize
   ┌────────────────────────────────────────────┐
   │  the sequence so far, as tokens            │ ◀─────────┐
   │  "The"   " Rasp"  "berry"  " Pi"   " is"   │           │
   └────────────────────┬───────────────────────┘           │
                        ▼                                   │
   ┌────────────────────────────────────────────┐           │
   │  FORWARD PASS - every layer, every weight  │           │
   └────────────────────┬───────────────────────┘           │
                        ▼                                   │
   ┌────────────────────────────────────────────┐           │
   │  a score for EVERY token in the vocabulary │           │
   │                                            │           │
   │  " a"       ██████████████████  31%        │           │
   │  " the"     ███████████         19%        │           │
   │  " small"   ███████             12%        │           │
   │  " capable" █████                8%        │           │
   │  ...        ~150,000 others                │           │
   └────────────────────┬───────────────────────┘           │
                        ▼  sample one (temperature, top-p)  │
                  ┌───────────┐                             │
                  │    " a"   │                             │
                  └─────┬─────┘                             │
                        │                                   │
             end token? ┤                                   │
                 │      │                                   │
                 │      └── no: append and run it all again ┘
                yes
                 │
                 ▼
   "The Raspberry Pi is a small, capable computer."

That is the whole trick. No plan, no draft, no lookahead - just the next token, over and over, each one conditioned on everything written so far. To see it happening, install the Python bindings and stream the output:

python3 -m venv ~/llm-venv && source ~/llm-venv/bin/activate
pip install llama-cpp-python

import time
from llama_cpp import Llama

llm = Llama(
    model_path="./models/qwen2.5-0.5b-instruct-q4_k_m.gguf",
    n_ctx=512,      # context window - bigger costs RAM
    n_threads=4,    # one per Pi core
    verbose=False,
)

prompt = "The Raspberry Pi is"

# Look at the tokens first - they are word *fragments*, not words
tokens = llm.tokenize(prompt.encode("utf-8"))
pieces = [llm.detokenize([t]).decode("utf-8", "replace") for t in tokens]
print("token ids :", tokens)
print("token text:", pieces)

# Now watch them arrive one at a time
start = time.time()
count = 0
for chunk in llm(prompt, max_tokens=60, stream=True):
    print(chunk["choices"][0]["text"], end="", flush=True)
    count += 1

secs = time.time() - start
print(f"\n\n{count} tokens in {secs:.1f}s = {count / secs:.1f} tok/sec")

Two things are worth pausing on when you run this. The first is the tokenizer output: the model does not see words, it sees fragments, which is exactly why LLMs are famously bad at counting the letters in a word. The second is the rhythm of the streaming - that pause between tokens is a full forward pass through every layer of the network, happening on a £60 computer.

What to expect

Rough throughput on a Pi 5 with 4-bit models and four threads. Measure your own - cooling, storage and model choice all move these numbers:

For comparison, a datacentre GPU serves the same 8B model at hundreds of tokens per second. That gap is the entire economics of the AI industry in one table.

Things that will bite you

• Swapping. If the model does not fit in RAM the Pi will not politely refuse - it will swap to disk and slow to something like a token per minute. If it seems hung, run free -h and pick a smaller model.
• Thermal throttling. vcgencmd measure_temp while it runs. Sustained inference will hit the throttle point without a fan or heatsink.
• Context length costs RAM too. The n_ctx value allocates a cache for every token in the window. Doubling it is not free.
• Small models confabulate confidently. A 0.5B model is a toy for understanding mechanics, not a source of facts. That is a feature for this exercise - it makes the difference between generating plausible text and knowing things impossible to ignore.