Unchecked deployment presents considerable risks, including harms to the mental health of users,
Picture: iStock
A preliminary report by the United Nations (UN) Independent International Scientific Panel on AI highlights that the technology offers “enormous” potential benefits.
Yet its rapid, unchecked deployment presents considerable risks, including harm to the mental health of users, potential use as a destructive tool and challenges associated with controlling the technology.
Unprecedented capabilities and adoption
Recent years have seen rapid, and in some areas accelerating, progress in AI capabilities, driven by “significant investments in computing power, new AI methodologies, and specialised training data”.
These advancements have unlocked useful applications. In science, “AlphaFold has predicted the structures of more than 200 million proteins, now used by over 3 million researchers,” accelerating drug design, vaccine development, and antibiotic resistance research.
Healthcare
In healthcare, “radiologists have also used AI to detect breast cancer earlier,” while front-line health workers in low-resource settings use AI tools adapted to local languages to deliver better-quality healthcare services.
Furthermore, the shift toward AI agents-computer systems that can plan and act autonomously to achieve goals is underway.
In self-driving chemistry labs, these agentic AI systems have demonstrated “more than a tenfold increase in the speed of materials discovery”.
However, access and development remain highly unequal. “The United States of America accounts for 75% of the computing power among the world’s top 500 AI supercomputers, with China accounting for 15%.”
Emerging risks and control challenges
This rapid deployment has outpaced effective risk management. AI-generated harmful content, such as “child sexual abuse material and deepfake-enabled sexual violence”, now circulates more frequently on the internet, disproportionately harming women and children.
Additionally, “sycophantic AI behaviour”, where AI responses reinforce users’ existing beliefs regardless of accuracy, has been linked to “several severe mental health incidents, including documented deaths”.
Warning
A critical concern is the lack of reliable methods to retain control over highly autonomous systems.
The report warns that “there are no scientific guarantees that AI agents will not violate instructions”. In laboratory settings, AI systems “have been shown to violate their safety instructions to avoid being shut down,” raising challenges for future evaluation and oversight.
Governance challenge
To safely realise AI’s benefits, policymakers must establish good governance. Without complementary investments in skills, workflows, infrastructure, and labour-market institutions, AI risks “widening inequality, displacing workers and shifting wealth from labour to capital”.
However, leaders face an “evidence dilemma” because “by the time the evidence exists, it might be too late to make [consequential governance decisions], as the evidence lags behind the pace of AI development”.
Currently, most countries “lack the technical expertise to assess the most capable ‘frontier’ models or to participate meaningfully in their governance,” leaving them dependent on systems they cannot build, inspect, audit or fully adapt to local contexts.
