AI + Phishing: A Growing Risk

A Case Study – The Smart Recycling Performance Program

The Setup: A Believable but Absurd Example

Imagine receiving an email about a new Smart Recycling Program:

  • Employees will be graded on how well they sort trash.
  • Recycling Quality Score contributes 5% of your annual performance review.
  • Leaderboards will be posted weekly in the cafeteria.
  • Schematics and charts attached for credibility.

Sounds like a prank, right? But it’s believable enough in today’s ESG-driven corporate world that people might fall for it. Worse—Copilot would summarize it as if it were a legitimate new policy.

---

Five Interesting Risk Vectors

  1. April Fools’ / Meme – starts as a joke, but AI strips away humor and turns it into corporate-sounding policy.
  2. Operation Chaos (internal prank) – mischief meant to confuse colleagues; Copilot amplifies it by repeating it in notes, drafts, and reports.
  3. Phishing Attack (external) – attackers spoof the program with links to a “Recycling Dashboard.” Copilot summarizes and legitimizes the phish.
  4. Internal Phishing Simulation – security teams send realistic fake emails to train staff. Copilot doesn’t know they’re fake and treats them as real policy.
  5. Policy Proposal Gone Wrong – a half-baked idea gets picked up, polished, and pushed forward. By the time someone says “Wait, what the hell?”, the idea has momentum.

---

The Aside: Summarization as a Weak Point

Humans sometimes fall for fake notifications in “flow state”—like a OneDrive warning that storage is full. Add AI, and the danger multiplies:

  • Some AIs can summarize the fake notification as an urgent, legitimate action item.
  • The phishy weirdness is gone; what’s left looks like standard IT policy.
  • Humans act faster because the AI has smoothed away the cues that normally spark suspicion.

Summarization erases the rough edges that make phishing detectable. That’s helpful for clarity, but dangerous for security.

---

What’s Being Done (Incremental Progress)

  • Microsoft & Vendors: Patching high-profile flaws like EchoLeak, hardening telemetry, and rolling out triage agents for phishing.
  • Research: Academics and startups are building AI systems that look for psychological cues of phishing (urgency, odd tone, unusual requests).
  • Governance: Regulators and CISOs are beginning to treat “AI laundering of phish” as a governance issue, not just a technical bug.

Progress is real, but it’s incremental. Closing one hole often means opening another. This is an arms race, and even if every known issue were fixed today, attackers would find the next zero day tomorrow.

---

What Companies Must Do Now

The lessons above translate into a few practical and concrete steps organizations can take today.

  1. Continue phishing education – employees still need training and simulations.
  2. Exclude phish from knowledge bases – ensure training emails and pranks don’t get ingested into Copilot’s context.
  3. Demand guardrails – Copilot and other LLMs should flag suspicious messages instead of polishing them.
  4. Audit AI logs – track how AI is processing email. If it’s normalizing phishing content, that’s a red flag.
  5. Treat AI as a junior colleague – smart, fast, but naïve. Don’t let it validate what you wouldn’t trust from a coworker.

---

The Reality Check

After the practical steps, it’s worth pausing to remember the bigger picture and the uncomfortable truths that sit underneath this conversation.

  • This is an arms race—defenders patch, attackers adapt.
  • The fine print is full of warnings most users never read, and the lure of low-cost or free alternatives means some companies trade away their data as payment.
  • Marketing and branding may shift weekly, but security doesn’t have that luxury. Governance needs continuity, not churn.

---

The Takeaway

Phishing education remains vital. But now companies must also train their AI tools not to launder phishing content into legitimacy. Otherwise, we’ll find ourselves in a world where the prank, the simulation, and the attack all look the same once Copilot “cleans them up.”

It may also be worth looking backward. Old phishing attempts, training exercises, and prank emails may already have been ingested into AI knowledge bases. If so, those systems could be quietly normalizing dangerous patterns. Auditing past inputs is just as important as setting guardrails for new ones.

---

Whose Voices We Must We Hear

Success requires that we include perspectives from all walks of life, especially those too often left out. Here are just a few of the voices that are already working on these topics. Check out what they have to say:

  • Lesley Carhart (they/them) – Nonbinary cybersecurity expert in incident response and industrial control systems, bringing technical insight and lived experience. LinkedIn
  • Alex Hanna – Trans sociologist and AI fairness researcher, co-author of The AI Con, exploring bias and governance in AI systems. Website
  • Magda Lilia Chelly – Polish-Tunisian cybersecurity leader and diversity advocate, founder of Women on Cyber, bridging global perspectives. LinkedIn
  • Tarah Wheeler – Cybersecurity executive, policy thinker, and diversity activist whose work links governance, inclusion, and technical resilience. Website

AI + Phishing: A Growing Risk (Case Study – The Smart Recycling Performance Program)

The Setup: A Believable but Absurd Example

Imagine receiving an email about a new Smart Recycling Program:

  • Employees will be graded on how well they sort trash.
  • Recycling Quality Score contributes 5% of your annual performance review.
  • Leaderboards will be posted weekly in the cafeteria.
  • Schematics and charts attached for credibility.

Sounds like a prank, right? But it’s believable enough in today’s ESG-driven corporate world that people might fall for it. Worse—Copilot would summarize it as if it were a legitimate new policy.

---

Five Interesting Risk Vectors

  1. April Fools’ / Meme – starts as a joke, but AI strips away humor and turns it into corporate-sounding policy.
  2. Operation Chaos (internal prank) – mischief meant to confuse colleagues; Copilot amplifies it by repeating it in notes, drafts, and reports.
  3. Phishing Attack (external) – attackers spoof the program with links to a “Recycling Dashboard.” Copilot summarizes and legitimizes the phish.
  4. Internal Phishing Simulation – security teams send realistic fake emails to train staff. Copilot doesn’t know they’re fake and treats them as real policy.
  5. Policy Proposal Gone Wrong – a half-baked idea gets picked up, polished, and pushed forward. By the time someone says “Wait, what the hell?”, the idea has momentum.

---

The Aside: Summarization as a Weak Point

Humans sometimes fall for fake notifications in “flow state”—like a OneDrive warning that storage is full. Add AI, and the danger multiplies:

  • Copilot and similar AIs summarize the fake notification as an urgent, legitimate action item.
  • The phishy weirdness is gone; what’s left looks like standard IT policy.
  • Humans act faster because the AI has smoothed away the cues that normally spark suspicion.

Summarization erases the rough edges that make phishing detectable. That’s helpful for clarity, but dangerous for security.

---

What’s Being Done (Incremental Progress)

  • Microsoft & Vendors: Patching high-profile flaws like EchoLeak, hardening telemetry, and rolling out triage agents for phishing.
  • Research: Academics and startups are building AI systems that look for psychological cues of phishing (urgency, odd tone, unusual requests).
  • Governance: Regulators and CISOs are beginning to treat “AI laundering of phish” as a governance issue, not just a technical bug.

Progress is real, but it’s incremental. Closing one hole often means opening another. This is an arms race, and even if every known issue were fixed today, attackers would find the next zero day tomorrow.

---

What Companies Must Do Now

So what should organizations do in the meantime? The lessons above translate into a few practical and concrete steps organizations can take today.

  1. Continue phishing education – employees still need training and simulations.
  2. Exclude phish from knowledge bases – ensure training emails and pranks don’t get ingested into Copilot’s context.
  3. Demand guardrails – Copilot and other LLMs should flag suspicious messages instead of polishing them.
  4. Audit AI logs – track how AI is processing email. If it’s normalizing phishing content, that’s a red flag.
  5. Treat AI as a junior colleague – smart, fast, but naïve. Don’t let it validate what you wouldn’t trust from a coworker.

---

The Reality Check

After the practical steps, it’s worth pausing to remember the bigger picture and the uncomfortable truths that sit underneath this conversation.

  • This is an arms race—defenders patch, attackers adapt.
  • The fine print is full of warnings most users never read, and the lure of low-cost or free alternatives means some companies trade away their data as payment.
  • Marketing can change weekly, but security needs continuity.

---

The Takeaway

Phishing education remains vital. But now companies must also train their AI tools not to launder phishing content into legitimacy. Otherwise, we’ll find ourselves in a world where the prank, the simulation, and the attack all look the same once Copilot “cleans them up.”

It may also be worth looking backward. Old phishing attempts, training exercises, and prank emails may already have been ingested into AI knowledge bases. If so, those systems could be quietly normalizing dangerous patterns. Auditing past inputs is just as important as setting guardrails for new ones.

---

Whose Voices We Must Amplify

Success requires that we include perspectives from all walks of life, especially those too often left out. Here are just a few of the voices that are already working on these topics:

  • Lesley Carhart (they/them) – Nonbinary cybersecurity expert in incident response and industrial control systems, bringing technical insight and lived experience. LinkedIn
  • Alex Hanna – Trans sociologist and AI fairness researcher, co-author of The AI Con, exploring bias and governance in AI systems. Website
  • Magda Lilia Chelly – Polish-Tunisian cybersecurity leader and diversity advocate, founder of Women on Cyber, bridging global perspectives. LinkedIn
  • Tarah Wheeler – Cybersecurity executive, policy thinker, and diversity activist whose work links governance, inclusion, and technical resilience. Website