Inference for Dummies: How AI Actually Generates a Response

For my non-technical folks that want to understand how inference works and what goes on behind the scenes when they send a prompt.

1. Introduction: What Happens When You Ask ChatGPT a Question?

When you type a question (a prompt) into the input box (chat), a number of actions are executed before the response is sent back to you. High level, the following layers are touched between the prompt and the response:

Text input → The text is broken down into smaller chunks (tokens) → The chunks are passed through multiple layers of a neural network to understand context, identify patterns, and determine what should come next → The model calculates probabilities for possible next tokens and selects one based on those probabilities → The selected token is added to the output, and the process repeats until the response is complete → Response is sent back to the user.

For example: You type in "What is the weather like in Paris?" This text is broken down into tokens, such as "What", " is", " the", " weather", " like", " in", " Paris", and "?". These tokens are passed through the neural network, which processes their relationships and context to determine what you're asking. The model calculates probabilities for the next token based on the input and selects one. It repeats this process, generating one token at a time, until the response is complete. The generated response is sent back to you.

Some technical terms which will be useful to familiarize yourself with:

• Tokenization: "The text is broken down into smaller chunks (tokens)." This process is called Tokenization.
• Neural Networks: The "layers of networks" are referred to as Neural Networks. More specifically, they are the layers that make up the neural network architecture.
• Inference: The process of using a trained model to generate an output from an input is called Inference.
• Autoregressive Generation: "The process repeats until the response is complete." This is called Autoregressive Generation (also called Autoregressive Decoding).

2. Training vs. Inference: Learning vs. Using What You've Learned

Before we dive deeper into how inference works, it's important to understand the distinction between two fundamental processes in machine learning: Training and Inference. Think of it like preparing for an exam. Training is the process of learning new information, studying patterns, and developing knowledge. Inference, on the other hand, is applying what you've already learned to answer a question or solve a problem that you might or might not have seen.

In the context of AI:

• Training: The process through which a model learns patterns and relationships from large amounts of data.
• Inference: The process of using an already-trained model to generate predictions or responses based on new input.

Naturally, you can see why the response quality inherently depends on the quality of the data (making good quality data today’s most prized commodity).

What Is Model Training?

Model training is the process of teaching an initially untrained model to recognize patterns, relationships between entities, and structures within data. Initially, the model has randomly initialized parameters and has not yet learned the patterns that allow it to generate useful responses. Through training, we expose the model to enormous amounts of data and adjust its internal numerical values (parameters) to improve its predictions. Over time, the model becomes increasingly better at recognizing patterns and generating accurate and relevant responses. This transformation of an untrained model into a trained model is called Model Training.

For example: During model training, if you asked the model to complete the sentence:

“The capital of France is ____”.

The model attempts to predict the missing word. Initially, it might predict something incorrect, such as London. Through training, it learns to improve its predictions by adjusting its internal numerical values. A model doesn't learn by someone manually correcting every incorrect answer. Instead, it uses a mathematical process to measure how far its predictions are from the expected answers and adjusts its internal parameters accordingly. Think of it like taking a test, receiving a score, identifying where you went wrong, and adjusting your approach before taking the test again. This process happens millions or billions of times across training data, gradually improving the model's ability to recognize patterns and make predictions.

There are several interconnected processes involved in training a model. Some of the most important ones are:

• Forward Propagation: The process of passing input data through the neural network to produce a prediction.
• Loss Function: A mathematical function that measures how far the model's prediction is from the expected answer.
• Backpropagation: The process of calculating how much each parameter contributed to the prediction error.
• Gradient Descent: An optimization algorithm that uses the calculated gradients to adjust the model's parameters and reduce the loss.

What Is Inference?

Inference is the process of using an already-trained model to generate predictions or outputs based on new input. Think of it like taking an exam after studying. During training, the model learns patterns and relationships from data. During inference, it applies what it has learned to answer questions or complete tasks it hasn't necessarily encountered before.

When you send a prompt to ChatGPT, you're initiating inference. The model takes your input, processes it using its learned parameters, and generates a response. Unlike training, inference does not involve updating the model's parameters. It's simply using what the model has already learned.

Some terminology worth knowing:

• Inference: Using a trained model to generate predictions or outputs from new input.
• Prompt: The input or instruction provided to the model.
• Parameters: The internal numerical values a model learns during training and uses to generate predictions.

3. Tokens and Embeddings

When you type something into ChatGPT, you see ordinary human-readable text. However, the model doesn't process words in the same way we do. It works with numerical representations of text. To understand how this works, we need to introduce two concepts: tokens and embeddings.

What Are Tokens?

A token is a small unit of text that a language model processes. A token can be:

• An entire word.
• Part of a word.
• A punctuation mark.
• A combination of characters.

For example, the sentence:

"What is the weather like today?"

Might be broken down into tokens such as:

["What", " is", " the", " weather", " like", " today", "?"]

The important thing to understand is that tokens aren't necessarily equivalent to words. A single word can consist of multiple tokens. Language models process text as sequences of these tokens rather than entire sentences or paragraphs.

What Are Embeddings?

Computers work with numbers, not words that we understand. This is where embeddings come in. An embedding is a numerical representation of a token, expressed as a list of numbers called a vector. These numerical representations allow the model to capture relationships between different words and concepts.

For example, the words "king" and "queen" are related concepts. Their embeddings can reflect similarities in meaning, while words like "car" or "banana" would have different representations. This is important for it to learn relationships and patterns between concepts. Think of embeddings as a way of translating human language into a mathematical format (model’s language) that a model can work with.

4. Neural Networks and Transformers: How AI Understands Context

Now that we've converted text into numerical representations through embeddings, we need to understand how a model actually processes that information. This is where neural networks come in.

What Is a Neural Network?

A neural network is a computational system made up of interconnected layers that process information and learn patterns. You can think of it as a series of processing stages. Information enters the network, passes through multiple layers, and is transformed along the way until the model produces an output. Each layer performs mathematical operations on the information it receives, allowing the network to identify increasingly complex patterns and relationships.

What Is a Transformer?

Not all neural networks are built the same way. Modern language models like ChatGPT are built using an architecture called a Transformer. Introduced in the 2017 research paper Attention Is All You Need, Transformers changed how machines process language. One of the biggest innovations was a mechanism called Attention, which allows the model to determine how different parts of the input relate to one another.

What Is Attention?

Consider the following sentence:

"The bank was next to the river, so we sat on its shore."

When processing this sentence, the model needs to understand that the word "bank" refers to the side of a river, rather than a financial institution. It also needs to understand how the surrounding words help establish that meaning. Attention allows the model to weigh the relevance of different tokens in relation to one another, helping it understand context rather than treating every word independently. This is particularly important because the meaning of a word can change depending on the sentence in which it appears.

In simple terms: Neural networks provide the structure for processing information, Transformers provide the architecture used by modern LLMs, and attention helps them understand relationships between tokens. With this context established, we can finally look at how the model uses everything it has processed to generate an answer.

5. Next-Token Prediction: How AI Generates a Response

At its core, a Large Language Model (LLM) works by predicting what token is most likely to come next, given everything that came before it. This is called Next-Token Prediction. For example, consider the sentence:

"The capital of France is..."

Based on the patterns it learned during training, the model might assign the following probabilities to possible next tokens:

The model calculates probabilities for possible next tokens and selects one based on those probabilities. According to the table above, "Paris" is overwhelmingly likely to be selected.

Does the Model Always Pick the Most Likely Token?

Not necessarily. LLMs can use a process called Sampling, which allows them to select from multiple possible tokens based on their calculated probabilities. This introduces some variation into responses, meaning the model doesn't necessarily produce the exact same wording every time.

One setting that influences this behavior is called Temperature.

• Lower temperature: Makes the model's selections more predictable, generally favoring higher-probability tokens.
• Higher temperature: Allows more variation by making lower-probability tokens more likely to be selected.

For example, if you ask ChatGPT to explain what a database is, a lower temperature might produce a straightforward, predictable explanation. A higher temperature might produce a more varied or creative explanation. Temperature doesn't make a model more knowledgeable. It changes how it selects its next tokens.

Ultimately, the model isn't retrieving a pre-written answer from a database. It's generating text by repeatedly predicting what should come next based on the context it has received and the tokens it has already generated.

6. Autoregressive Generation: How AI Builds a Complete Response

We've established that an LLM predicts one token at a time. But how does it turn these individual predictions into a complete sentence, paragraph, or response? This process is called Autoregressive Generation. Rather than generating an entire response at once, the model generates one token, adds it to the existing sequence, and then uses the updated sequence to predict the next token.

For example: Imagine you ask ChatGPT:

"What is 2 + 2?"

The model might generate its response through the following sequence:

After each prediction, the newly generated token is added to the existing sequence. The model then uses the updated sequence to predict what comes next. This continues until the model reaches a stopping condition and the response is complete. As you can see, the model isn't generating the entire sentence at once. It's building the response incrementally, one token at a time.

7. Putting It All Together: What Happens When You Send ChatGPT a Prompt?

Let's revisit our original question:

"What is the weather like in Paris?"

Now that we've covered the fundamentals of inference, we can understand what happens behind the scenes.

1. Input: You enter your question into ChatGPT. This is your prompt.
2. Tokenization: The text is broken down into smaller units called tokens.
3. Embeddings: These tokens are converted into numerical representations that the model can process.
4. Context Processing: The numerical representations pass through the Transformer's neural network, where attention helps the model identify relationships and context.
5. Next-Token Prediction: The model calculates probabilities for possible next tokens and selects one.
6. Autoregressive Generation: The selected token is added to the response, and the model repeats the prediction process until the response is complete.
7. Output: The completed response is converted back into human-readable text and displayed in ChatGPT.

And that's inference! :) What appears to be a simple exchange between you and ChatGPT is actually a series of mathematical operations happening behind the scenes.