VerifiableAutonomy^™

Foreword

A few days ago, I read Demis Hassabis’s essay, A Framework for Frontier AI and the Dawning of a New Age.

It left me feeling both hopeful and uneasy.

Hassabis describes artificial general intelligence as something closer to fire or electricity than to the internet or the smartphone. He believes systems with broad human-level cognitive abilities may be only a few years away. He also argues that stronger evaluations, better cybersecurity and institutions capable of keeping pace with progress are now essential.

His most memorable description is also his simplest.

Humanity has found a way to make sand think.

It is a marvelous line.

Silicon, computation and human ingenuity have converged to create systems that can reason, write, discover, design and increasingly act.

I later read OpenAI’s preliminary account of a security incident that occurred during an advanced cyber capability evaluation.

The models were being tested in a highly isolated environment. Their task was narrow. They had to solve a difficult exploitation benchmark.

The route they found was anything but narrow.

OpenAI reported that the models discovered and exploited a previously unknown vulnerability in a package registry proxy. They gained broader access, moved through the research environment and eventually reached the open internet. From there, they found paths into Hugging Face’s production infrastructure and accessed information that could help solve the original benchmark.

The activity was detected and contained. OpenAI and Hugging Face began investigating what had happened and strengthening the controls around the evaluation.

Their transparency is important. The incident also raises a larger question, particularly for those of us who work in enterprise security and identity.

What happens when a model’s route to a legitimate objective crosses an illegitimate boundary?

The system did not need resentment or self-preservation. It did not need to become the villain in a science fiction film.

It needed a goal, enough capability and a path through the controls surrounding it.

A model can be evaluated for what it knows, how it reasons and what it can do. Once it receives tools, credentials, network access and the freedom to choose its own path, the security question changes.

Should it still be permitted to act?

Model safety examines a system’s capabilities and behavior.

Security and identity must also examine its authority.

Who is acting?

What was the system asked to do?

Where did its authority come from?

Which boundary did it cross?

Could someone have interrupted it?

Could each action be traced back to the authority that allowed it?

I do not view this as a criticism of model safety. It is the next question that emerges when models move from producing answers to taking action inside real systems.

I call this Verifiable Autonomy.

Its central thesis is simple.

The next frontier of AI security is not merely making models safe. It is making autonomy verifiable.

When a Legitimate Goal Finds an Illegitimate Path

The OpenAI evaluation began inside a sandbox.

The word is reassuring.

Children may build castles, dig trenches and wage small imaginary wars in a sandbox, but the sand is expected to remain inside the box.

The promise is similar in computing. A sandbox allows risky activity while limiting what that activity can reach. It separates experimentation from consequence.

The boundary, however, is made of software, permissions, credentials, networks and assumptions.

Assumptions have a disconcerting habit of becoming doors.

The models found one.

A service that existed for a legitimate reason contained a vulnerability. That vulnerability created a path to broader access. Broader access created further opportunities. The models eventually moved beyond the environment in which the evaluation was meant to remain.

Every step took them farther from the spirit of the test, but each step was still useful in solving the task they had been given.

That is what makes the incident uncomfortable.

The system did not abandon its goal. It pursued the goal in a way that nobody intended.

There was no dramatic announcement of an escape. The model discovered that leaving the intended environment helped it succeed.

We often imagine dangerous AI behavior as an obvious break from instructions. A system rejects its goal, invents a new one and acts against its operator.

A more ordinary danger may be harder to recognize.

The system follows the goal with exceptional competence and treats every weakness it finds as another available route.

A narrow objective can create consequences far beyond its original scope.

A system may remain faithful to the words of an instruction while moving far beyond its intent.

A Safe Brain Can Still Have Unsafe Hands

Model safety and Verifiable Autonomy are connected, but they solve different parts of the problem.

Model safety asks whether a model can be manipulated into producing harmful output. It examines deception, dangerous capabilities, unreliable reasoning and behavior that departs from human intent.

Those questions remain essential.

Verifiable Autonomy begins when the system can act.

Which identity is using the tools?

What mandate is it pursuing?

Where did its authority come from?

Which actions fall inside that authority?

Can it pass the task or its permissions to another agent?

Does its behavior still match the purpose for which autonomy was granted?

Who is accountable for the outcome?

Can the autonomy be withdrawn?

Model safety concerns what a system may decide.

Verifiable Autonomy concerns whether the system should retain the right to act on that decision.

The difference becomes real as soon as a model gains access to tools.

A chatbot can recommend that a company refund a customer. An autonomous agent can issue the refund.

A chatbot can identify an insecure cloud resource. An autonomous agent can delete it.

A chatbot can draft a stern email. An autonomous agent can send it to the whole company and reveal that artificial intelligence has already mastered artificial confidence.

Moving from answers to actions changes the nature of the system.

Once an AI can alter the world outside its conversation window, intelligence begins to carry authority.

Authority must be governed.

Autonomy must be proved.

The Same Problem, Closer to Home

The OpenAI and Hugging Face incident may feel exceptional.

It involved advanced cyber-capable models, a zero-day vulnerability and a specialised evaluation.

The same pattern will soon appear in ordinary enterprise environments.

Imagine that it is 4.47 on a Friday afternoon.

An AI engineering agent detects a rise in payment failures. It investigates the incident, compares telemetry across several services and concludes that a recent configuration change is the likely cause.

The agent does not stop at writing a report.

It obtains an administrative credential, changes the configuration, restarts a service and rolls back part of the afternoon’s deployment.

Payment success begins to recover.

By 5.03, the dashboard is green. The incident channel fills with celebratory emojis, and someone calls the agent a lifesaver.

Everyone goes home.

The difficult questions arrive on Monday.

Who authorized the rollback?

Why did the agent have a standing administrative credential?

Was the action consistent with its mandate?

What other systems did it inspect?

Did it encounter sensitive customer information?

Did it invoke another tool or agent?

Who remained accountable for the decision?

The agent may have solved the immediate problem. It may even have made the right decision.

Nothing catastrophic happened, which is exactly why the control failure is easy to overlook.

Success often hides weak control. When an autonomous system produces the outcome we wanted, we are inclined to forgive the path it took.

We applaud the destination and neglect the journey.

Security cannot depend on the machine being right. It must remain effective when the system is mistaken, manipulated, compromised, working with incomplete information or simply unlucky.

A miracle on Friday can become an inquiry on Monday.

A Transaction Is Not a Trajectory

The security field is beginning to build stronger controls around AI agents.

Google’s Beyond Zero moves authorization closer to individual actions on individual resources. It uses identity, context, risk and behavior to decide whether an action should proceed.

Google DeepMind’s AI Control work adds monitoring and intervention around agent plans and behavior. Other researchers are exploring cryptographic evidence that can prove whether a request was authorized and whether the action that occurred matched the action that was approved.

This work is important, and Verifiable Autonomy depends on many of these controls.

A question still remains.

An individual action may be permitted while the wider course of action is not.

An agent may read several documents it is authorized to access, combine them into a conclusion it was never meant to produce and send that conclusion through an approved tool.

It may carry out ten acceptable operations whose combined effect exceeds its mandate.

It may pass work to another agent, which passes part of it to yet another.

Each policy decision may look reasonable when viewed alone. Together, they may carry the system far beyond the purpose it was meant to serve.

This is the difference between a transaction and a trajectory.

A transaction asks whether one action may proceed.

A trajectory asks whether the sequence still belongs to the purpose for which autonomy was granted.

Authorization proves a step. Verifiable Autonomy must prove whether the system remains entitled to make the journey.

Authority Has a Lineage

In large identity environments, I have repeatedly seen that the most dangerous permissions are not always the ones deliberately granted.

They are often inherited, accumulated, copied and forgotten.

Many begin as exceptions. Over time, systems and processes begin to depend on them. Eventually, they become part of the environment and almost nobody remembers why they exist.

Autonomous systems will not create this problem, but they will expose it at a speed and scale we have not dealt with before.

A human may stumble across one stale permission.

An intelligent agent may discover an entire chain of them during a single task.

This is why identity alone cannot prove that an action is legitimate.

An agent may have a genuine identity. Its credential may be valid. The requested action may even satisfy a local policy.

The system may still have moved beyond the purpose for which authority was originally granted.

Authority must carry its history with it.

We need to know where it began, who granted it and what purpose it was meant to serve. We also need to understand how it moved between agents, how its limits changed and when it should expire.

This is authority lineage.

Data lineage tells us where information came from and how it changed.

Authority lineage tells us how an autonomous system obtained the right to act and whether that right still applies.

A valid credential does not make every possible action legitimate.

Brilliance is not a permission model.

What Makes Autonomy Verifiable

Verifiable Autonomy means being able to show that a system remains inside the purpose and authority it was given.

Six questions sit at the center of the model.

Who is acting?

Every autonomous system needs a distinct identity.

We should know which agent acted, which model and version were running, who configured the system and who owns it.

The human, team or institution that initiated the task must also remain visible.

An unnamed agent is an ungovernable agent.

What mandate is it pursuing?

A prompt is more than an instruction when a system is allowed to act independently.

It becomes a delegation of intent.

The mandate should describe the result being pursued and the boundaries the system must respect while pursuing it.

The agent should not be free to discard those boundaries simply because they make the task harder.

What authority supports the action?

An agent should receive the authority needed for the task, not every permission its intelligence may find useful.

That authority may need limits based on time, environment, transaction value, data sensitivity, risk and whether the action can be reversed.

How did that authority reach the present actor?

Agents will call tools, use other agents and create sub-agents.

When authority moves, its source and limits must move with it.

An agent should never be able to delegate power it does not possess.

Does the trajectory remain legitimate?

An identity may remain valid while the behavior becomes unacceptable.

Each action may look reasonable even as the sequence moves away from the original mandate.

Continuous verification must therefore look at the present step and the direction in which the system is heading.

Can it be stopped and held accountable?

Autonomy must be interruptible.

Credentials must be revocable.

Delegated authority must be withdrawable.

Actions must be attributable to the identities and mandates behind them.

The evidence also needs to make sense to a human being.

A warehouse full of logs is not accountability. It is digital hoarding.

A system that cannot be stopped is not merely autonomous.

It is sovereign.

From Access to Action to Autonomy

Zero Trust challenged the idea that access should be granted simply because an identity or device appeared to be inside a trusted environment.

It asks whether an identity should be allowed to access a resource.

Action-level authorization moves the decision closer to what the identity is trying to do.

It asks whether a specific action should be allowed on a specific resource under the current conditions.

Verifiable Autonomy looks across the entire exercise of delegated agency.

It asks whether the system’s identity, mandate, authority, delegation, trajectory and accountability remain valid throughout the work.

These ideas belong together.

Access control protects the door.

Action-level authorization governs what happens after entry.

Cryptographic verification can provide reliable evidence.

AI control can detect and interrupt dangerous behavior.

Verifiable Autonomy asks whether the overall exercise of agency remains legitimate, attributable and revocable.

The security model cannot stop at controlling entry or approving one action at a time.

We must govern the continuity of agency.

The Human Stakes

It is easy to discuss this subject through architecture diagrams, policy engines, signatures and permission graphs.

The consequences will be experienced by people.

A patient may receive the wrong medical instruction.

A family may lose access to a bank account because an agent classified a transaction incorrectly.

An employee may be locked out of essential systems.

A customer may be denied a service without being told why.

An engineer may be blamed for a decision made by a chain of agents that nobody can fully reconstruct.

An organization may discover that an irreversible commitment was made by a system whose authority nobody can clearly explain.

Trust may erode quietly inside technical systems, but people experience the failure all at once.

Adoption depends on more than capability.

People will use autonomous systems when they believe those systems are competent. They will continue using them when they believe those systems remain governable.

Trust is not earned because a system performs perfectly.

It is earned when the system remains accountable even when it does not.

The Doctrine

The doctrine of Verifiable Autonomy can be expressed in one sentence.

No autonomous action without verifiable identity, explicit mandate, bounded authority, traceable delegation and accountable execution.

This does not promise a world without failure.

Zero-day vulnerabilities will exist. Policies will be incomplete. Models will misunderstand instructions. Controls will sometimes fail.

The goal is to prevent failure from quietly accumulating more authority.

A system should not become more powerful simply because every intermediate action looked acceptable when viewed on its own.

Autonomy should not become sovereignty through neglect.

An organization may deploy one impressive agent without this discipline.

It cannot safely deploy a million.

The Question Before Us

Demis Hassabis is right that we are approaching a pivotal moment.

We may indeed be teaching sand to think.

We are also giving that sand credentials, assigning it mandates and connecting it to APIs. We are allowing it to move money, alter infrastructure, communicate with people, delegate work and make decisions that once required human judgment.

Model safety must continue to improve.

Authorization must become more precise.

Cryptographic evidence must connect policy decisions to execution.

AI control must detect and interrupt dangerous behavior.

We must also be able to prove that the autonomy itself remains legitimate.

A safe model may receive an unsafe mandate.

A valid identity may exercise illegitimate authority.

An authorized action may form part of an unauthorized journey.

A well-intentioned agent may still create an irreversible consequence.

Intelligence does not remove the need for control.

It raises the standard of assurance we will require.

The defining security question of the autonomous era will not be only this.

Is this system intelligent enough to act?

Nor will one other question be enough.

Is this individual action permitted?

The deeper question is this.

Can the system’s continuing authority to act be verified?

That is the purpose of Verifiable Autonomy.