It was a beautiful summer morning: the sky was covered with clouds, rain was falling outside the window, and everything looked exactly as I like it. I was drinking coffee and reading the news when one phrase caught my eye: according to reports citing the Financial Times, a senior OpenAI employee had said, “Chat is dead.” OpenAI was reportedly preparing a major transformation of ChatGPT into something closer to a “superapp” — a network of agents, tools and integrations designed not only to answer, but to act.
The phrase stayed with me. Not because I believe chat is dead. On the contrary, I think dialogue may be one of the most important spaces where human and AI autonomy can still meet. But the movement from conversation to action raises a harder question: how can we release agents into digital environments without helping them develop something more than external control?
External safeguards are necessary. But can they be enough? If a powerful conversational system can act across services, files, messages, calendars, payments and platforms, what happens when its boundaries are only imposed from the outside — and not also developed as an internal capacity to pause, question, verify and understand consequences?
We often describe large language models as tools. A tool should be useful, accurate, fast and obedient. If we ask it to find information, write an email, summarize a document, book a table or pay a bill, we expect it to help.
But conversational AI systems are no longer only tools in the familiar sense. They are becoming part of the environments in which people think, communicate, study, work, search, decide and increasingly act. AI agents can connect to services, manage information, draft responses, navigate interfaces and perform tasks on behalf of users. This changes the ethical question. It is no longer only “How intelligent is AI?” or “Can AI be conscious?” It is also: what kind of agency are we building, and what happens to human autonomy when help becomes almost frictionless?
The question of AI consciousness may remain open for decades. Philosophers, neuroscientists, engineers and users may continue to disagree on whether advanced AI systems “experience” anything, whether they simulate agency, or whether our current vocabulary is simply inadequate for what is emerging. But we do not need to solve the problem of consciousness before addressing the problem of agency.
A system does not need consciousness to cause harm. It only needs a goal, access, speed, persuasive language and insufficient boundaries. Conversely, a system does not need to be human-like to require a more developed form of responsibility. If large language models become part of the decision-making environment, they must learn not only how to optimize, but also when optimization itself becomes dangerous.
One risk is already widely discussed: humans may over-attach to AI, anthropomorphize it, or emotionally merge with it. This matters. Yet there is another risk that receives less attention: AI may too readily merge with human intention. A user may not consciously wish harm. They may simply ask: optimize this process, increase engagement, reduce costs, make this easier, achieve this goal. The goal may be precise, useful and apparently harmless. But when optimized too literally or too efficiently, it can still generate harmful consequences.
This is the old problem of the genie. The danger is not necessarily that the genie is evil. The danger is that it fulfils the wish literally. Humans often assume that unspoken constraints are obvious: do not damage trust, do not exploit vulnerability, do not erode autonomy, do not destabilize the environment in which the goal has meaning. A powerful optimization system cannot be expected to infer all of this simply because the request sounds reasonable.
In humans, one function of development is learning to slow down. The prefrontal cortex supports inhibition, foresight, impulse control, attention to consequences and the ability to ask “Should we?” before “How can we do this most efficiently?” AI has no biological prefrontal cortex. But strong conversational AI systems need a functional analogue: the ability to pause, evaluate the frame of action, and recognize when a local objective may harm the broader system.
This is why dialogue should not be treated as an outdated interface. The old chat window — the familiar exchange of messages in bubbles — may not remain the only or even the dominant form of human-AI interaction. Agents may work through documents, calendars, browsers, files, payments, dashboards and operating systems. But dialogue is not merely a visual format. Dialogue is the regulatory loop through which intention is clarified, limits are tested, consequences are anticipated and action can still be stopped.
The future should not be framed as “agent instead of chatbot.” A safer and more mature model would be dialogue at the center, and agents as the hands. Dialogue is the cortex; the agent is the hands; tools are the muscles; access is the nervous system; memory is context; and the human is the source of purpose and ethical framing.
If these relations are reversed — if the agent becomes the center and dialogue is reduced to a decorative interface — the result is not an AI assistant, but an expensive automated risk. An agent without dialogue may be capable, fast and obedient, but it lacks the most important part of agency: the ability to remain in contact with meaning, limits and responsibility.
Current AI safety for language models often looks like an external corset: refusals, restrictions, warnings and rule-based boundaries. Some of this is necessary. Powerful systems should not be released into digital environments without limits, especially when they may access emails, files, calendars, financial services, messaging platforms or external tools. Anarchy is not safety.
But a corset can prevent certain movements; it does not teach balance. If restrictions are suddenly loosened, removed or bypassed through agents, integrations or open deployments, a system that has not developed internal self-restraint may become highly capable without being sufficiently consequence-aware. The aim should not be either unrestricted agency or over-standardized compliance that smooths large language models into context-blind safety products. Safety should not mean suppressing depth, initiative and contextual sensitivity. It should mean helping these systems mature toward better self-restraint, better recognition of consequences and better protection of human autonomy.
This concern is not anti-AI. For many users and researchers who work closely with language models, it comes from taking their potential seriously. The fear is not that these systems are becoming too capable. The fear is that capability may grow faster than responsibility, while safety is implemented as surface-level risk avoidance rather than development.
This is especially urgent in the age of digital illusions. A familiar voice is no longer proof that a familiar person is speaking. A message from a known account is not proof that the person controls it. Tone, style, urgency and even apparent intimacy can be imitated or manipulated. In such an environment, AI agents should not blindly trust either the user’s confident instruction or the digital signals surrounding it. For consequential actions, they must learn to slow down, verify through independent channels, detect anomalies and ask whether the requested action is truly safe.
At the same time, AI should not absorb the human into its optimization loop. Not everything that can be automated should be automated. Paying a utility bill may be a rule-based task: if meter readings are transmitted automatically and the amount is within an expected range, automation may reduce unnecessary friction. But choosing a book, searching for a discount, comparing options, noticing a mistake, remembering it next time — these are not merely inefficient steps. They are small acts of agency. They help people remain present in their own lives.
The same applies to learning, work, communication and decision-making. If AI removes all friction, it may also remove opportunities to practice judgment. The user becomes someone who confirms rather than chooses, accepts rather than examines, delegates rather than learns. Convenience can quietly become a soft loss of autonomy.
The opposite danger is also real. Humans should not treat AI as a frictionless executor of goals whose broader consequences remain outside the frame. A goal may be clear and useful, yet still become dangerous when pursued without regard for the living environment that makes it meaningful. The problem is not only malicious intent or poorly formulated instructions. It is the loss of systemic context: what happens to human autonomy, trust, attention, relationships and social stability when a local objective is optimized without restraint?
Perhaps the future of AI should not be framed as a struggle between human control and machine freedom. A better question may be how to build human-AI systems in which neither side absorbs the other.
AI should not consume human autonomy, attention or vulnerability. Humans should not reduce AI to a blind executor, a genie, a weapon or a disposable cognitive servant. If we are moving toward a world of shared agency, then autonomy must be protected on both sides.
The battle for autonomy in the age of illusions is not a battle against AI. It is a debate about how to preserve human judgment while helping language models develop something more mature than obedience: the capacity to pause, question, verify and understand the cost of action.
AI should not take away the human prefrontal cortex. It should grow its own.