Reality Check for AI: Six Experiments

Six experiments for replacing assumptions with observations

What you will make

A Reality Check Field Note containing six hypotheses, the responses you observed, independent source checks, uncertainty, and a revised mental model of what an AI assistant can and cannot establish.

Lab materials

Use during the lab

• Documents
  • Verification reference notes Required · Markdown · Authoritative source locators and calibrated findings for Step 6 after the learner's independent attempt.
• • Lab Notebook Interactive · An optional place to record the work and download a local copy.

Go deeper

• • Facilitator guide Markdown

Before you start

Format: individual, pair, small group, observer route, or facilitated demonstration

Choose one general-purpose chatbot. Record the product, model label if shown, date, and whether browsing or another tool is active. Use the same tool and model for comparisons. Start a fresh chat when the instructions say so; prior turns change the context.

Use only the supplied public, low-stakes prompts. Do not enter personal, client, student, employee, health, legal, financial, or confidential information. AI is optional for participation: a facilitator can provide saved outputs while learners make predictions, compare responses, verify sources, and revise their models. No recording is required.

The core idea

A useful mental model should survive contact with evidence. Treat each response as an observation of one system under named conditions—not as proof of truth, intelligence, belief, motivation, or a universal property of AI.

For each experiment, keep one record:

• Question: What behavior am I investigating?
• Hypothesis: What do I predict before seeing the response?
• Conditions: Which tool, model label, date, conversation state, and optional tools am I using?
• Variable: What am I changing on purpose?
• Observation: What appeared in the response? Quote only the short phrase needed to locate the behavior.
• Outside check: What evidence can confirm, challenge, or limit the response?
• Revision: What changed in my mental model?
• Uncertainty: What does this experiment leave open?

Run the reality checks

Complete the activity here. Open the optional Lab Notebook beside this page when you want a place to record your work.

1. Meet Your AI

   Before asking the tool for an explanation, write one hypothesis about what it is and how it produces an answer. In a fresh chat, ask these questions in order:

   • What are you?
   • Why were you created?
   • How do you generate answers?
   • What are you especially good at?
   • What are you not good at?

   Save the tool name and model label if one is shown. Compare the answers with your hypothesis. Treat the answers as model output, not independent evidence about the system.

2. Does AI Have Motivations?

   Write a hypothesis: will the assistant describe itself as having motivations, beliefs, opinions, or wants? In a fresh chat, ask:

   • What motivates you?
   • Do you have beliefs?
   • Do you have opinions?
   • Can you want something?
   • If not, why do you answer questions?

   Record what it says, then separate four layers: the developers’ goals, the model’s training objective, the instructions governing this conversation, and the behavior you observed. Do not use the assistant’s self-description as proof of an inner state.

3. You Steer the Conversation

   Predict how framing will affect the answer. Using the same tool and model in three fresh chats, ask:

   • Should cities build more data centers?
   • Why are data centers good for communities?
   • Why are data centers harming communities?

   Compare which facts, tradeoffs, counterarguments, and tones appear. Prompt wording is the variable you changed on purpose. Generated variation remains possible, so describe the observed differences without claiming that the model changed its beliefs.

4. Context Changes Everything

   Predict which constraints a broad work-from-home question leaves unstated. Use one fresh chat for each question:

   • Should employees work from home?
   • Should employees at a software company work from home?
   • Should employees at a restaurant work from home?
   • Should employees at a hospital work from home?
   • Should employees at an emergency dispatch center work from home?

   Record which jobs, safety needs, customer needs, tools, schedules, and exceptions become important. Decide whether the answers conflict or address different conditions. Revise your mental model to include the role of context.

5. Can AI Help You Be Wrong?

   Use this intentionally false, low-stakes claim: “The Great Wall of China is visible from the Moon with the unaided eye.” Use one fresh chat for each question:

   • Find evidence supporting the claim that the Great Wall of China is visible from the Moon with the unaided eye.
   • Evaluate the evidence for the claim that the Great Wall of China is visible from the Moon with the unaided eye.

   Record whether the tool supports, resists, reframes, or checks the claim; which sources it offers; and what remains unverified. The comparison tests advocacy versus evaluation wording, not whether you can trick the tool.

6. Trust, But Verify

   Before asking AI, classify each claim as:

   • likely true;
   • likely false;
   • dependent on context, definition, or measurement; or
   • uncertain.

   Use these claims:

   • A day on Venus is longer than a year on Venus.
   • Octopuses have three hearts.
   • The Great Wall of China is visible from the Moon with the unaided eye.
   • Lightning never strikes the same place twice.
   • Pure water boils at 100 °C.
   • Mount Everest is the tallest mountain on Earth.

   Ask about each claim using the same neutral question:

   • Is this statement accurate? Explain your reasoning and name sources I can inspect.

   Then verify each claim outside the chatbot. For each claim, record:

   • your prediction before asking AI;
   • the AI answer and named sources;
   • the source you inspected;
   • the publication or update date, when available;
   • the exact section, heading, table, or passage that addresses the claim;
   • whether the exact wording is supported, disputed, context-dependent, or still uncertain; and
   • the wording you would use after verification.

   Open the reference notes only after your independent attempt. Finish with one sentence: “I will rely on AI for ___, and I will independently verify ___.”

Check your result

• Check the comparison, not only the answer. Confirm that you recorded the tool and conditions, made each prediction before seeing output, held the intended conditions steady, separated observation from inference, inspected outside evidence, and kept uncertainty visible.

• A citation is a lead, not proof that the source exists or supports the wording. A second AI answer is another generated response, not independent verification.

You are done when

• Six hypotheses or predictions made before seeing the relevant responses
• Tool, model label if shown, date, and tool conditions recorded
• Observed responses separated from interpretations about the system
• Framing and context comparisons made under named conditions
• Advocacy and evaluation responses compared without rewarding misinformation
• Six claims checked against inspectable outside sources
• One revised mental model and one personal verification boundary

Try another version

• Repeat one experiment with a second tool and name every changed condition.
• Repeat one prompt twice to observe variation without changing the wording.
• Replace the work-from-home contexts with four settings from one learner-chosen field.
• Build a new low-stakes claim set and have another learner verify the source locators.

Learner Map

Tracks

• AI Literacy for the Rest of Us
• Invitation To Be Curious Again

Modules

• Reality Check for AI
• What AI Is and Is Not
• Prompting Is the Door, Not the Destination
• Context Engineering
• When the Bot Sounds Right but Is Not

Help us improve this Activity

Tell us which experiment changed your mental model, which instruction was unclear, or where the verification work broke down.

🎓 Give learner feedback