What Is a Large Language Model (LLM)?

TL;DR

• A large language model (LLM) is a neural network trained on huge amounts of text so it can predict and generate language. Chatbots such as ChatGPT, Claude and Gemini are built on LLMs [2].
• It writes a reply one small piece of text, called a token, at a time, each time picking a likely next token.
• Training has stages: learning from lots of text, then being taught to follow instructions and to give answers people prefer [2].
• It can sound confident and still be wrong, so important facts need checking.

What an LLM is #

A language model estimates how likely a piece of text is to come next in a longer passage. Think of finishing the sentence "I ___ in my kitchen": some words are far more likely than others [1]. A large language model does this at huge scale, with billions of adjustable numbers called parameters and a very large amount of training text. Most modern LLMs use a design called the transformer, whose attention mechanism lets the model relate every part of the text to every other part, however far apart they are [2].

How a chatbot generates text #

Models work with numbers, not letters, so your message is first cut into tokens: words, parts of words or single characters. In English a token is roughly four characters, or about three quarters of a word [1]. The model then repeats a simple loop:

1. Read all the text so far, as tokens.
2. Work out how likely each possible next token is.
3. Pick one, either the most likely or a random choice among the likely ones [1].
4. Add it to the text and go back to step 1.

A reply is just this loop running until the model produces an end marker. Because there is some randomness in the pick, asking the same question twice can give different wording.

How an LLM is trained #

None of this stores a neat list of facts. The model learns statistical patterns in text, which is why it can write fluently about almost anything.

What LLMs do well and badly #

LLMs are good at drafting, summarising, translating, explaining and answering questions in natural language [2]. They are weaker when accuracy matters and the answer is not well covered in their training text. Biased or inaccurate training data can make output less reliable, and researchers still treat LLMs largely as black boxes whose inner workings are hard to read [2]. They also can only use the text that fits in their context window, the amount of text they can consider at once [3].

FAQ #

Is ChatGPT the same as an LLM? #

Not exactly. ChatGPT is a chatbot product, and an LLM is the technology inside it. Claude and Gemini work the same way, each built on its own models.

Does an LLM understand what it writes? #

It is trained to predict text, and sources describe its skill as pattern prediction rather than human-style understanding. Whether that counts as understanding is debated, so it is safest to treat its answers as drafts to check.

Where does an LLM get its information? #

From patterns in its training text. Some chatbots can also look things up or read documents you give them, which adds fresh information to the context window [3].

Why do answers change when I ask again? #

Many chatbots pick among likely tokens instead of always the single most likely one, so wording can vary between runs [1].

Sources #