AI Is Only Magic When We Close Our Eyes

Recent reports about frontier AI systems have generated dramatic headlines. We read about containment breaches, unexpected paths around safeguards, and systems communicating in ways their human operators did not anticipate.

Before we decide that an AI escaped, there is a more immediate question: What kind of system did we build around it?

The reports prompting this reflection include controlled evaluations and a real infrastructure compromise. Those are not the same thing, and neither should be minimized. Yet both direct our attention beyond the model. Humans defined the objectives, chose the tools, configured the environments, granted permissions, and decided how the work would be monitored.

The model found paths through a system that humans built, configured, and were responsible for supervising.

The system is bigger than the model

It is tempting to talk about “the model” as though it were a single actor. In practice, we are building systems composed of interacting parts:

• objectives and incentives
• models and tools
• memory and shared state
• permissions and security controls
• monitoring and escalation paths
• human operators and organizational processes

Any one component may work as designed while the interactions among them produce an unexpected or harmful outcome. Complex systems rarely fail because of one rogue component. More often, reasonable decisions combine in ways no one intended.

Acknowledging the surrounding system does not make the incident less serious. It identifies the part senior leaders can govern.

It is only magic when we close our eyes and wait for the reveal.

Scale changes the conversation

AI systems can make decisions, retry approaches, and adapt far faster than people can follow. At machine scale, low-frequency failure modes become recurring operational concerns. Human attention does not scale at the same rate.

Other high-speed systems have already taught us this lesson. Manufacturing, aviation, finance, and networking all had to develop ways to observe complex systems and intervene before small problems became large ones.

We should expect an AI system with tools, memory, permissions, and repeated opportunities to find combinations its designers did not predict. The specific path may surprise us. The class of behavior should not.

If we had been watching closely enough, with the right visibility and enough people who understood the full system, much of what looked mysterious afterward might have been understandable while it was happening.

Autonomy is a leadership decision

The decision to grant autonomy belongs at least at the senior-leadership level. When the system can reach consequential data, infrastructure, customers, money, or public trust, that decision may belong in the C-suite.

Delegating the work does not transfer accountability. Leaders decide what the system may reach, how much independent action it may take, what evidence they need before expanding that authority, and who can stop it when the evidence changes.

They also decide whether the organization has enough capacity to understand what the system is doing. That means funding visibility and monitoring, defining escalation paths, and giving the people responsible for oversight enough authority to act.

Autonomy is not a switch. It is a series of increasingly consequential leadership decisions. An organization should not grant more autonomy than it can responsibly absorb.

New capability may require new work

Leadership often begins by fitting a new technology into roles it already understands. That can be a good first step, but it can hide work that no current role was designed to perform.

Industrial systems came to rely on safety engineering. Commercial aviation relies on air traffic control. Networked organizations came to need cybersecurity. Advanced AI may create its own operating functions: people who supervise, audit, validate, and safely integrate autonomous systems into organizations.

That may become a new profession. It may become a responsibility distributed across several professions. Either way, leadership must recognize the work before assigning it to people whose existing jobs are already full.

The cost of oversight is part of the product

This leads to a practical question for senior leaders: If an AI system cannot economically support the level of oversight necessary for safe operation, is it ready for that level of deployment?

Organizations have long used versions of this readiness test for factories, aircraft, power plants, pharmaceuticals, and other technologies capable of producing tremendous benefit alongside meaningful risk.

Safety has always been part of the product. Oversight belongs in the business case.

An invitation

AI now does more than answer questions. It operates inside systems built from software, infrastructure, incentives, organizations, and people.

Senior leaders do not need to understand every technical detail. They do need to understand the system well enough to decide how much autonomy the organization can support, what new capabilities it must build, and who remains accountable when the system surprises them.

Senior leaders must ask both: What can the model do? and What must our organization become ready to do around it?

The encouraging part is that these are still human decisions. We remain in the driver’s seat. Now we need to do the driving.