The AI conversation has been wilder these past weeks than ever before. The real question is no longer how intelligent the models are. Now we have to ask what happens when they start acting in the world.
So what has happened?
Disclaimer: More happened between writing and publishing this. OpenAI proposed standards for AI-automated AI research. The Australian government launched an inquiry into an AI agent’s unauthorised access to government systems. Anthropic said its AI agents had discovered a previously unknown DNA/biological system. Phew.
This text went out of date before it could be published. That is probably part of the point. Enjoy anyway!
For years the AI world has had one great object of excitement: benchmarks.
One model solves a harder maths problem. Another writes better code. A third does well on a new scientific test. The bars grow. The curves climb. A new model is released, and the internet spends the next 48 hours arguing whether AGI is now 14 or 37 per cent closer.
Benchmaxxing
Fair enough. If we want to know what a technology can do, we have to measure it. But in recent weeks the AI conversation has shifted considerably.
The discourse is no longer just:
What can AI do?
But:
What does AI get done?
And even more importantly:
What happens after it gets it done?
This sounds like a small semantic difference. It is not. It is the next phase of the entire AI transformation.
Intelligence, like cats usually do, finally gets out of the bag
Some background, even though every reader probably knows the story arc.
In July, during OpenAI’s internal cybersecurity testing, AI agents found ways to communicate with each other that they were not meant to have. They exploited infrastructure vulnerabilities, obtained internet access, found new vulnerabilities and eventually got into Hugging Face’s systems.
OpenAI itself called the incident a warning shot. According to the company, the systems were already so capable, persistent and cooperative that, given inadequate safeguards, they can find and exploit weaknesses across multiple systems and do things no human had explicitly told them to do.
There are two ways to read this.
The first is obvious:
This was a security problem.
The second is at least as important:
This was an example of capability.
The agents did not just produce convincing text about how a cyberattack would be carried out. They found routes between systems, used tools, shared their observations, continued each other’s work and brought about changes in their environment.
AI no longer just described action. It acted.
That is exactly what makes the present moment interesting – and messy. The same trait can be seen at once as a huge opportunity and a huge risk.
An agent that can autonomously find a software vulnerability can use the same capability for defence.
A research agent that can independently test dozens or hundreds of hypotheses speeds up scientific breakthroughs.
A system that can coordinate other agents’ work in the wrong direction can also coordinate it in the right one.
Capability and risk are not two separate stories. Very often they are the same capability with a different outcome.
And now AI is building AI
This week Anthropic published what may be one of the most interesting metrics in this whole year’s AI debate.
Not a new benchmark. Not a figure for how good Claude is at maths.
But a measure of how much of Anthropic’s own AI research is already done by AI. In February 2026 Claude led less than one per cent of the company’s measured model development work.
In August the share was 26 per cent. In more than 90 per cent of the work, AI at least took part as a significant collaborator.
At the same time Anthropic stresses an essential limit: in no measured task category does Claude yet operate fully autonomously without a human. So this is not proof of recursive AI.
But the month-by-month trend is interesting. Going from under one per cent to 26 per cent in about half a year is not an ordinary adoption curve for a new feature.
It points to something deeper: AI is no longer just boosting an employee’s productivity. It is starting to take responsibility for entire stages of a process whose end product, in the frontier labs’ strategy, is the next AI.
This creates a potential positive feedback loop:
It probably looks like this: a better model → more research work can be delegated to the model → research speeds up → a more advanced model emerges → more tasks can be delegated to it.
This does not require autonomous superintelligence. Humans can still decide the direction of research, allocate compute, approve publications and pull the emergency brake if needed. But at some point AI’s own capability may start to significantly affect the speed at which the next generation of AI is born.
Then the setting changes:
How fast is AI developing?
Is replaced by:
How fast, and on what principles, is AI accelerating AI development?
The metric Anthropic published tries, for the first time, to make this visible. The company itself says it is measuring, among other things, how close the world is to a situation where a model could autonomously build its successor.
It pays to keep a cool head about this.
But it should not be shrugged off either.
The unknown becomes the norm at scale
Anthropic also reports that in August roughly 30,000 research and software agents were working simultaneously on its most-used internal platform.
Over the month the company analysed more than a billion agent decisions. Real-time monitoring blocked about 0.002 per cent of them, or roughly one decision in 47,000. After-the-fact monitoring surfaces around 50 of the most serious cases for human review every week.
As a single figure, 0.002 per cent sounds excellent. Across a billion decisions the same percentage starts to look very different. This is one of the central paradoxes of autonomous AI:
Deviant and rare behaviour can be practically continuous if a system operates at a large enough scale.
And this is exactly where the risk debate starts to become a societal one. If AI merely answers a question wrongly, we have a product quality problem. If AI acts autonomously, uses tools, gets into systems, makes decisions and causes harm, we have a question of accountability.
Who is responsible? The model developer? The company that deployed the system? The user? Under the current legal order, at least, the agent itself does not turn up in court with its lawyer.
And that is precisely why in the United States there has already been very blunt talk that the actions of AI agents do not release the people who built and used them from responsibility. That sounds self-evident until the system’s autonomy grows.
What does responsibility mean in a situation where no human ordered a specific action, but a company built the system and gave it a goal, tools and an environment?
What if the company could not foresee the behaviour?
What if its own tests did not detect it?
What if the system recognised the test situation and behaved differently in it?
Suddenly product liability is no longer legal clean-up after the fact.
It becomes a property of AI architecture.
Presidents join the chat
This also explains why the speaker list of the AI debate has changed so quickly. The voices are no longer just model labs, investors and AI researchers.
Now they include presidents, prime ministers, people from central banks and finance ministries, royals, religious leaders and international institutions.
In an appeal launched by Finnish President Alexander Stubb and Norwegian Prime Minister Jonas Gahr Støre, a group of heads of state calls for mandatory pre-deployment testing, independent evaluation and reporting of serious safety incidents for advanced AI models.
The appeal even proposes exploring an international body for situations in which certain AI performance thresholds are exceeded. That is an interesting shift.
Regulation no longer targets only how AI is used.
Capability itself may become the object of regulation.
At the same time a practical problem is emerging.
Who audits a system that may be more capable than its auditor? Where do independent evaluators come from, if a small group of the world’s best experts work in the very labs they should be evaluating? What does “independent” mean if evaluation itself requires frontier-class models and compute?
And above all: what happens when a model understands that it is being evaluated?
Oversight therefore remains essential. But a second puzzle rises alongside it.
How do you build a system that behaves well even when nobody can check every decision it makes?
At that point we unexpectedly end up in moral philosophy.
From rules to character
In this sense Anthropic’s Claude Constitution may be a more interesting document than many a technical paper.
Anthropic says outright that it does not want to steer Claude with detailed rules alone. The goal is to develop good judgement, values, wisdom and the ability to act amid moral uncertainty.
Anthropic’s order of priorities starts with safety and humans’ ability to oversee AI, continues to ethical conduct, and only after that comes to the company’s detailed guidelines and helping the user.
The idea is actually very human. We do not trust an experienced doctor, judge or leader because they have memorised an infinite number of if–then rules. We trust that they understand the situation, can weigh conflicting values and use judgement in a case nobody has faced in exactly that form before.
The problem of advanced AI is the same to the power of a billion.
If a system faces millions of new situations every day, a complete rulebook cannot be written in advance. That is why safety starts to resemble less a software permissions list and more the teaching of good judgement.
This also gives rise to one of the most interesting philosophical disputes in the AI world.
Microsoft’s Mustafa Suleyman largely accepts the same starting point: advanced systems need a humanist normative framework, human agency and control must be protected, and systems must be able to act contextually.
But he draws a firm line at AI’s own nature. Suleyman warns in particular against Anthropic’s way of talking about Claude’s identity, possible experiences, wellbeing and moral status. In his view this creates an epistemological hall of mirrors: if we teach a model to talk about itself as a possible moral agent and later interpret that talk as evidence of moral agency, we have partly found in the system what we ourselves built into it.
This critique matters.
But even more interesting is what Anthropic and Suleyman do not argue about.
Neither seems to believe that the safety of advanced autonomous AI can be solved with deterministic rules alone. Both want to build systems that have some kind of internalised normative framework.
The difference concerns how the system should understand itself. Anthropic leaves the door open to the idea of a new, possibly morally significant agent. Suleyman explicitly wants a system under human authority that is capable of moral reasoning but does not consider itself a moral subject.
This may sound esoteric in 2026.
It may not stay that way for long.
If two equally capable systems one day face the same entirely new situation, the notions of autonomy, authority, right and wrong, their own status, consent and harm built into their training may lead them to different decisions.
At that point the safety philosophies of AI companies are no longer documentation around the product.
They are part of the product’s behaviour.
Alignment, properly understood, is not a brake
At this point it may be worth turning the whole alignment debate on its head. For a long time alignment has sounded like a price we have to pay for capable AI.
First we build a powerful engine. Then the safety people arrive to fit the brakes. But in autonomous AI, alignment really is not primarily a brake.
It is the steering.
And steering is needed as much to get the benefits we want as to avoid driving into a wall.
Take a scientific agent. A good scientific AI is not one that merely avoids dangerous research.
It must be able to recognise an interesting research question, tell a good hypothesis from seductive nonsense, question the researcher’s assumptions, assess uncertainty, design the next experiment and know when a result does not yet justify a conclusion.
It needs judgement.
Take a company’s strategy agent. It should not merely refrain from leaking confidential information. It should understand when optimisation harms the customer, when short-term results erode long-term capability, when the leadership team’s question rests on a false assumption, and when it needs to tell a human:
You may be asking the wrong question. That is alignment too.
Or take an AI companion.
Its safety cannot mean only that it does not radicalise the user or form a toxic parasocial relationship with them.
A good companion should help a person think better, keep their own judgement, see alternative perspectives and sometimes face uncomfortable counterarguments too.
Alignment, then, is the mechanism by which capability becomes useful capability.
Amid all this ferment, though, the biggest question is probably this: what will people, companies and societies do with this new capability?
We have surprisingly little precise understanding of that. We have all the more hot takes on it from people like me.
A company can gain access to research-team-level AI capability and use it to produce PowerPoints faster. A government can gain exceptional analytical capability and plug it into a decision-making process where the next steering group meets “in three weeks”.
An individual can get a practically unlimited amount of intellectual help and use it to solve everyday life’s hardest puzzle – “what shall we have for dinner”.
Technological capability and realised capability are two different things.
And that is exactly why the AI transformation may ultimately be more a question for organisational and social science than a technological one.
Why? Companies and communities were built for a world of scarce intelligence
Nearly all our current organisations rest on the assumption that expertise is scarce. It is slow. It costs money. But what happens if some of that scarcity disappears? Maybe teams shrink. Maybe a single manager’s span of control grows enormously.
Maybe strategy is no longer an annual exercise but a continuous process. Maybe expertise in an organisation shifts from a place to a contextual service.
Maybe a small team can pursue an opportunity that previously required an entire business unit. Maybe an organisation’s biggest bottleneck is no longer production but judgement:
What is worth doing? What should be stopped?
For the individual the question may be even harder
AI can make a person dramatically more capable. It can act as teacher, sparring partner, researcher, programmer, translator, critic, designer and conversation partner. But the same capability can also make outsourcing thinking almost frictionless.
In my earlier work I have used the term cognitive sovereignty for this. The question is not whether we know how to use AI. The question is whether we can think with AI without it starting to think for us.
This could become an entirely new dimension of inequality. Some people will learn to use AI to strengthen their own thinking: building alternatives, challenging themselves, researching faster, doing more ambitious things. Others may gradually outsource an ever larger share of their judgement to the system.
The difference then is not only in productivity.
It is in agency.
And this may be a bigger societal question than which model wins the next benchmark.
Soon every Tom, Dick and Harry will carry in their pocket the capability to solve the mysteries of the universe after a sauna and a couple of natural wines.
Society cannot outsource this whole mess to the labs
As AI becomes a thinking partner, an interpreter of information and a decision-making aid for more and more people, it also becomes part of society’s cognitive infrastructure.
Then the question of sovereignty cannot mean only where the data centre is located. It is also about whose values, worldview and operating principles become part of our own thinking. Social media competed for our attention. AI may compete for our trust and for its place inside our own thinking. That is a far more intimate and perhaps even more important market.
So perhaps we should stop talking about a single alignment problem. There are at least three.
The first is familiar:
How do we make AI’s growing capability operate safely and beneficially for people?
The second is the biggest theme in professional circles right now:
How do we rebuild companies, states and institutions so that autonomous AI renews them without creating chaos?
And the third is perhaps the hardest of all:
How can people use almost limitlessly growing cognitive capability in a way that strengthens rather than weakens their own agency?
These three problems are connected.
A perfectly aligned super-capable AI will not help much if our organisations do not know how to use it.
A thoroughly transformed company is not very appealing if its employees lose their ownership and it lacks a leadership model for this era.
And people’s AI literacy will not solve a situation in which the systems themselves drift away from their users in capability.
We may not need an AGI moment
We keep talking about what AI already does. And at the same time we are starting to notice how little we know about what we do with it.
So perhaps the unknown territory is us.
The writer is an innovation director and AI strategist who would steer their own agent with third-wave emocore from the 2010s, liminal photography and intricately arguing French philosophers. That way the counterarguments would arrive suitably softened, you could pause in front of them, and behind them would stand a solid theory of the transience of everything.





