Lesson Objective
Understand Token-Based Language Prediction
A large language model does not think like a person. It processes text as tokens, converts those tokens into numerical vectors, studies the surrounding context and then predicts what token is likely to come next.
Audio Lesson
Listen to This Lesson
The audio version provides a deeper discussion of large language models, exploring how they are trained, how they predict the next token in a sequence, how they maintain context during conversations, and why they have become one of the most important technologies in modern artificial intelligence.
Concept Overview
LLMs Predict Language One Token at a Time
When you ask an AI model a question, the model does not search for a pre-written answer. Instead, it analyses the prompt, compares the token relationships inside its trained neural network and generates a response step by step. Each new token becomes part of the context for the next prediction.
Learning Algorithm
LLM Text Generation Workflow
| Step | Process | Technical Meaning |
|---|---|---|
| Step 1 | Receive prompt | The model receives a user question, instruction or conversation history. |
| Step 2 | Tokenise input | The text is split into smaller units called tokens. |
| Step 3 | Create embeddings | Tokens are converted into numerical vectors that the model can process. |
| Step 4 | Use context window | The model considers the available prompt, previous conversation and current token sequence. |
| Step 5 | Apply attention | The model gives stronger weight to the tokens that matter most for the next prediction. |
| Step 6 | Predict next token | The model calculates a probability distribution and selects a likely next token. |
| Step 7 | Generate response | The model repeats the prediction process until a complete answer is formed. |
| Step 8 | Review limitations | The user should check for missing context, bias, outdated information or hallucinated details. |
Step 1
Receive the Prompt
Every interaction begins with a prompt. The prompt may be a question, command, document extract, code sample or previous conversation. This input tells the model what task it should attempt and provides the context that guides the rest of the generation process.
Technical Point
The prompt provides the immediate task and the context the model uses to begin generating a response.
Step 2
Tokenise the Input
The model cannot process ordinary text directly. The prompt is first broken into tokens, which may be whole words, parts of words, numbers, punctuation marks or symbols. Tokenisation turns language into manageable units for the model.
Technical Point
Tokenisation converts human language into smaller units that the model can process mathematically.
Step 3
Create Embeddings
After tokenisation, each token is converted into a numerical vector called an embedding. Embeddings help the model represent meaning, similarity and relationships between words. For example, tokens connected to learning, intelligence and reasoning may sit closer together in the model’s mathematical space.
Technical Point
Embeddings convert tokens into vectors so the model can process meaning and relationships numerically.
Step 4
Use the Context Window
The context window is the amount of text the model can consider at one time. It may include the user’s current prompt, earlier messages, uploaded text or previous generated tokens. If important information is outside the context window, the model may not use it.
Technical Point
The context window controls what information the model can consider when making the next prediction.
Step 5
Apply Attention
Attention helps the model decide which tokens are most important for the current prediction. Instead of treating every token equally, the model learns relationships between words, phrases and positions. This allows it to connect ideas across a sentence, paragraph or longer conversation.
Technical Point
Attention allows the model to focus on the most relevant token relationships before predicting the next token.
Step 6
Predict the Next Token
The model produces a probability score for many possible next tokens. It does not know the “correct” word in a human sense; it calculates which token is statistically likely based on the prompt, context and attention patterns. Generation settings can make the answer more predictable or more creative.
Technical Point
Next-token prediction is a probability calculation based on the available context and learned token relationships.
Step 7
Generate the Response
Once a token is selected, it is added to the output. The model then repeats the process: the new token becomes part of the context, attention is recalculated and another token is predicted. This loop continues until the model reaches a stopping point or completes the answer.
Technical Point
A response is built by repeatedly predicting, selecting and appending tokens until the answer is complete.
Step 8
Review the Limitations
LLMs are powerful, but they are not perfect. They can misunderstand the prompt, invent details, reflect bias in training data or miss information that is outside the context window. Important answers should therefore be checked against reliable sources, especially for technical, legal, financial or safety-critical work.
Technical Point
LLM output should be reviewed because the model can produce convincing text that is incomplete, biased or incorrect.
Key Takeaways
What You Should Remember
Tokens
LLMs process language as small text units rather than full human sentences.
Embeddings
Tokens become numerical vectors so the model can work with meaning mathematically.
Attention
The model weighs the most relevant token relationships before predicting what comes next.
Limitations
Generated answers still need checking, especially when accuracy matters.
Knowledge Check
Quick Large Language Models Quiz
Test your understanding. The questions can change when you refresh them.
Lesson Summary
Large Language Models Summary
Large language models generate text by converting prompts into tokens, representing those tokens as vectors, applying attention across the context window and predicting one token at a time until a complete response is produced.