Commentary
Chatbots Are Not Witnesses
We should test powerful tools aggressively. We should not mistake those tests for testimony.
Senator Bernie Sanders recently recorded a public conversation with Claude about privacy, power, and democracy.1 The exchange was worth watching because it made a difficult question visible: What can we actually learn when a fluent system is asked to describe the risks around it?
A chatbot’s apparent confession is still a generated continuation, not direct access to its internal reasoning or motives. An apology or admission may offer clues about how the system responds under a particular set of prompts, constraints, and conversational conditions, but it is not evidence of what caused any specific answer. The challenge is identifying those conditions well enough to understand what the response does, and does not, tell us.
I saw this while testing an AI “second brain” built from years of code, documentation, and data definitions. When I asked about assignments, the system pointed me to a downstream table named Assignment. When I reframed the question to ask which table was the authoritative source for assignment data, it identified the upstream core table instead. A third version, asking where an engineer should begin tracing the assignment data lineage, produced a more cautious answer that described the relationship between the two.
The system had not changed its beliefs between questions. Each phrasing activated a different interpretation of what I meant by “where the data is stored”: the most obvious table name, the system of record, or the beginning of the lineage. The test revealed that the first answer was unstable because the original question lacked an important constraint.
Test the behavior
When a tool may influence a consequential decision, testing is necessary. Vary the wording. Ask for sources. Press on a contradiction. Change the framing and see whether the answer holds.
Those tests can reveal instability, overconfidence, invented support, or a missing constraint. They can show that an answer which sounded settled on the first pass changes when the question is put another way. That is useful evidence about how the system behaves in use.
Fluency is not perception.
Why the format misleads us
An interview has a familiar shape. One party asks questions; the other answers in full sentences. A chatbot can sound reflective, certain, even sincere. The format invites people to treat the answer as a statement of knowledge.
A current chatbot does not have firsthand perception of a disputed event. It generates an answer from its model, the prompt, system instructions, retrieved material, and other context available to the exchange. Its response may be interesting, useful, or alarming. It is still an output produced under conditions.
That is why a chatbot cannot give testimony in the ordinary sense. A witness can be examined about what they perceived, remembered, understood, or had reason to distort. Those questions do not map cleanly onto a generated answer.
Records are different
This distinction does not make chatbot records irrelevant. Prompts, logs, outputs, and system records may matter in a dispute or an investigation. They can help establish what a company presented to a customer, what a user relied on, or what conditions surrounded an action.
Two well-known cases illustrate different risks. In Mata v. Avianca, lawyers submitted fictitious cases generated by ChatGPT and were sanctioned.2 The lesson was not that the chatbot testified falsely. Its output was treated as legal authority without verification. In Moffatt v. Air Canada, a tribunal considered information supplied through the airline’s website chatbot when resolving a customer dispute.3 The case concerned the company’s conduct through its channel, not the chatbot as an independent witness.
A note on the legal examples: I am an engineer, not a lawyer. These cases illustrate how chatbot outputs and records can be treated differently from testimony; they are not legal advice or a complete account of the law.
A useful rule
Run the experiments. Publish the transcripts when they show something important. Compare answers, test the framing, and keep asking where the system breaks down.
Then describe the result accurately: this is an output under these conditions. It may tell us something important about whether the tool is fit for a task. It does not tell us what the machine witnessed, intended, or believed.