How LLMs Answer Questions They Were Never Taught

The spark

As I started learning about agents, I asked my chatbot, in its hippie-nerdy personality, how language models know things. It replied with a cosmic story about language bending through a higher-dimensional dreamfield.

The image was memorable. It was not an explanation I could use.

Here is the more useful version: a language model is trained to predict tokens from the text that came before. In use, it produces the next token from the prompt, its available context, and its learned parameters. Transformer models use attention to weigh relationships within the input; that architecture helps explain why changing the context can change a response. (Attention Is All You Need)

That does not make the model a database, a witness, or a mind reading from a shelf. It means the model can often continue a pattern in a useful way, including on a question it never saw in exactly that wording.

A new question is not necessarily new work

Suppose I ask for five animals that cannot fly and are not birds. The exact sentence may never have appeared in the training material. Its parts are familiar: animals, flight, exclusion, lists, and ordinary language about them.

The model can combine those patterns and produce an answer. That ability is related to what researchers call in-context learning: a model can adapt its behavior from instructions and examples supplied in the current conversation without receiving a new round of training. Large language models have demonstrated useful few-shot behavior, but it is uneven and does not make every answer reliable. (Language Models are Few-Shot Learners)

The distinction matters. An answer can be novel in wording and still be wrong, outdated, incomplete, or based on an incorrect premise. Fluency is evidence that the system can generate fluent text, not that it has verified the underlying claim.

Why context changes the answer

Context changes the task the model appears to be solving.

If I first say, “Explain this to a fourth grader,” I have asked for different vocabulary and examples than if I say, “Write a field note for a database administrator.” If I include a policy, a meeting transcript, or a source document, I have given the model material it can use—along with the risk that it will misread, omit, or overstate that material.

Settings matter too. System instructions, available tools, retrieval results, model version, and sampling choices can affect an output. That is why a workplace assistant and a personal chatbot may feel different even when they use related underlying models. The difference is not proof that one has more or less knowledge; it may be a different context, configuration, or access boundary.

“Latent space” can be a helpful metaphor for learned internal representations. It is not a map we can inspect well enough to say that a particular prompt sent the model down a known geometric path. The metaphor should not replace an explanation of the system a reader is actually using.

A better way to investigate an answer

When an answer matters, I can treat it as a starting point for investigation rather than a confession from an inner mind:

1. Name the claim. What, exactly, is the model asserting?
2. Check the source. Ask for a primary source or locate one independently; do not treat a generated citation as proof.
3. Vary one condition. Change the wording, add a relevant document, or remove an irrelevant instruction and see what changes.
4. Separate the model from the system. Note which model, version, tools, retrieval sources, and instructions were available.
5. Keep a human decision owner. For consequential work, someone must decide whether the answer is adequate to use.

This is more useful than asking the model why it said something and accepting its explanation at face value. A model can generate a plausible account of its answer without providing a faithful account of the computation that produced it.

What this means for agent design

An agent is not a machine that has suddenly learned how to reason like an engineer. It is a system that combines a model with instructions, tools, memory or retrieval, and controls.

Good design makes those parts visible. It gives the agent a bounded task, reliable sources where needed, a way to record uncertainty, and a check before an output becomes an action. Clear prompts matter, but they are only one control in a larger system.

The interesting thing about a language model is not that it can answer a question it was never explicitly taught. It is that its answer invites a better question from us: What evidence, context, and review would make this answer safe enough to use?