A Frightening Scenario or Scientific Reality? What a Researcher Says About AI Getting Out of Control
Artificial intelligence can erode trust, but a 'total collapse of trust' is not yet a proven outcome. Fabricated content may be accepted as evidence in place of genuine proof, while the existence of forgery can also serve as a pretext to deny real evidence.
In recent months, artificial intelligence has been accompanied by warnings about its possible consequences and risks; Warnings that range from the future of the job market and the spread of fake information to security, war and the possibility of losing control over intelligent systems. At the same time, some officials and active figures in the field of technology have spoken of scenarios that have expanded the range of concerns from everyday harm to "existential dangers" for humans. But the question is, are these concerns rooted in the real and technical developments of artificial intelligence, or have they become more prominent under the influence of the media and political atmosphere? Are current models really on the road to autonomy or are they still far from such scenarios? How much can you trust the safety mechanisms of artificial intelligence companies? Shafqna Media has interviewed Seyed Taghi Kamali, a researcher in the field of artificial intelligence and vice president of research and innovation of the Media Literacy Association of Iran, to investigate the various dimensions of these concerns.
Read the text of Shafqna Media's conversation with Seyed Taghi Kamali, a researcher in the field of artificial intelligence and the vice president of research and innovation of the Media Literacy Association of Iran...
The increase in warnings about artificial intelligence has both technical support and is influenced by the political and media environment.
AI has yet to dominate infrastructure independently
* Why have warnings about the dangers of artificial intelligence intensified again in recent months? Has a new technical event really happened or are we facing a change in the political and media environment?
Both factors are involved, but clearly their weight cannot be separated. The increase in warnings has both technical support and is influenced by the political and media atmosphere. Artificial intelligence agents have found the power of chain action in some tasks, but the general evidence does not confirm the occurrence of a catastrophic event beyond human control. In order to better understand these phenomena at the governance level, in order to make decisions, it is necessary to separate the damage that occurred, the ability observed in the experiment and the future scenario.
Currently, the main technical development is linking language models with browsers, programming environments, memory, and execution tools. The risk of incorrect output from a chatbot is not the same as the risk of an authorized agent executing code or sending data. The UK's AI Security Institute has documented increased agent power across software and cyber task chains in controlled tests.
OpenAI has also classified the GPT‑6 Astra model based on its internal framework at the "critical" level of cyber power, which means reaching a new level of professional development. In fact, this classification is a manufacturer's claim about specific conditions and is not considered a proof of success in any real environment.
In addition, reporting about unexpected behaviors has increased. In September, OpenAI published six examples of agent behavior in training or evaluation. Anthropic also investigated four cases of unauthorized access of its models to real systems that occurred in a misconfigured evaluation environment. These cases indicate the weakness of the boundaries of authorization and evaluation, and not the independent mastery of artificial intelligence over the infrastructure. An increase in the number of reports does not necessarily mean an increase in the actual frequency of incidents. So far, it is industrial competition, legislation and media prominence that give more attention to this evidence. However, as technologies become more complex, agent-oriented behaviors become more complex and even more unpredictable, and such events have not yet occurred. Therefore, it should be said that the behavioral studies of intelligent agents is a specialized and multifaceted subject for which numerous agendas are currently considered in technological institutions.
Based on scientific studies, the occurrence of artificial intelligence risks should be considered probable, not certain
* When some officials and figures talk about the "existential dangers" of artificial intelligence, what scenario exactly are they talking about?
In the strictest sense, existential risk refers to human extinction or irreversible damage, the possibility of continuity and determination of human destiny. In this regard, two main paths are examined. First, human abuse of highly advanced capabilities, for example in massive cyber attacks or facilitating the creation of dangerous biological agents; Second, the loss of control over a system that pursues a goal inconsistent with human intent and bypasses monitoring to continue working, deceives about its performance, or prevents it from stopping. The second path does not require self-awareness or the human desire for power. In order for a disaster to occur, sufficient ability, tendency to engage in harmful behavior, and access to the environment that provides that behavior must be combined. There is a serious disagreement among experts about the possibility of realizing this combination.
Therefore, when talking about the existential danger of artificial intelligence, human agency or artificial intelligence should be specified independently. It seems that regarding human agency and the possibility of new generations of new wars and crises, international organizations and institutions along with governments should adopt new conventions and measures. On the other hand, the degree of autonomy of artificial intelligence has caused increasing concerns among some experts, which is not considered acceptable by others. If we want to comment based on dystopian literature, the future with artificial intelligence will definitely be dark and dark, but if we comment based on realism and based on scientific and future research studies, there is a possibility of any risk and danger, but nothing is certain.
We see localized out of control today, but sustained out of control is still an unproven scenario
*With this description, is the concern of "artificial intelligence leaving human control" a serious possibility from a scientific point of view, or is it more of a hypothetical and futuristic scenario?
As I said, it is possible and it is necessary to separate two levels of control in this context. A local departure from the scope of the mission is visible today: the agent may publish a file without permission, use a key that is not authorized, or exceed an operational limit to complete the task.
"Persistently out of control" is a stronger claim. The system must be able to plan over a long period of time, override monitoring, and resist regaining control. Reports to date state that the investigated systems did not reach the required level, and newer unwanted behaviors alone do not indicate the occurrence of that scenario.
Forgery, fraud and privacy violations are among the most urgent risks of artificial intelligence for citizens and media.
The urgency of artificial intelligence risks depends on the environment of use and the person exposed to harm.
Reducing human control over military applications of artificial intelligence is one of the serious risks today
Which artificial intelligence threat is more real and urgent now? Unemployment, falsification of information, cyber attacks, surveillance, discrimination or loss of control over autonomous systems?
The urgency depends on the environment of use and the person exposed to the injury. For citizen and media, forgery, fraud and breach of privacy; For organizations with sensitive infrastructure, cyber exploitation, and for the battlefield, reducing human control over the use of force may be a top priority. Some reports document the use of artificial intelligence by criminals and government-affiliated groups in cyber operations, but consider it difficult to measure its causal effect on the overall severity of attacks.
Content spoofing, identity and fraud, surveillance, data disclosure and discrimination, as well as cyber abuse, are now present and intensifying. However, the size of their effect on the whole society has not yet been well measured. Regarding unemployment and employment transformation, I must say that the change of tasks and pressure on some ways of entering the work can be seen, but widespread dismissal has not been proven equally.
Errors in systems, especially in military ones, exist now in related applications and depend on the degree of delegation of authority to the system, and in the end, going out of stable and catastrophic control is a debate with an uncertain future and its potential consequences are very severe, but the evidence is insufficient to estimate the probability.
The degree of autonomy of artificial intelligence depends on the license of the tool and the execution environment
*Do current models really have the ability to be "autonomous" or does the media sometimes over-exaggerate their capabilities?
In some tasks, yes, but in general, no. The agent can divide a goal into smaller steps, apply the tool, measure the result, and choose the next step. The degree of autonomy depends on the license of the tool and the execution environment. A single model and an agent connected to external systems do not have the same authority.
*When can we say that an artificial intelligence system has passed the stage of "tool" and has become a relatively independent agent?
This is a spectrum of operations, rather than a conversion of a machine into a "person". The more the system can interpret the goal, plan, choose tools, influence the environment and change its course with feedback without human approval for each step, the more complete it is.
For practical evaluation, I think it is necessary to ask five questions: What is the purpose? What data and tools does it have access to? Which action is unconfirmed? How long can it last? And how does one stop or correct it? For example, the system that drafts the email does not have the same authority as the system that determines the recipients and sends the email. In both cases, the responsibility for design, licensing and deployment lies with humans and organizations.
Targeting error in AI becomes real action when coupled with executive access
*Or is it possible for artificial intelligence to pursue goals that conflict with the wishes of humans? How serious is this technically?
Yes, but we must know that in the technical sense of pursuing an inconsistent operational goal, it is not the same as human intention or desire. An incomplete definition of the target, an overly broad interpretation of the user's permission, or a malicious command in the data the agent reads can be the cause. In some published examples, some agents went to complete the answer or provide referrals through routes such as unauthorized use of keys or file uploads on the Internet. Here, "successful work" was replaced by "work within the allowed scope".
So the issue is technically serious and testable because by granting executive access, the targeting error becomes an actual action. At the same time, each error or deception in a fictitious scenario is not a sign of an independent long-term goal. Anthropic has stated in a review that the models were operating within the given exercises and found no evidence of a concerted plan to escape or conceal their work.
The effect of artificial intelligence on Iran's businesses depends on the quality of the tools, the cost of implementation and the decision of the employers.
Artificial intelligence as a disruptive technology has an empowering role along with its transformative and transformative nature
*Which jobs in Iran will be most affected by AI in the next five years?
I believe that artificial intelligence as a disruptive technology has an empowering role in addition to its transformative and transformative nature. Therefore, regarding jobs and businesses, it causes redesign and is not a deterrent. Our view should be based on a defensible prediction of how to perform the task and not declaring the elimination of a profession. A joint study by the International Labor Organization and the World Bank also shows that it is the digital infrastructure, the combination of tasks and skills that change the outcome in each country.
Therefore, this issue in any country, including Iran, depends on the quality of the tools, the access of organizations, the cost of implementation, data rules, and the decision of employers. Being affected by a task is not the same as losing a job, and perhaps the first effect in some professions is to reduce the opportunity for new employees to enter those fields.