King Charles warns tech bosses of 'existential dangers' of AI

King Charles III warned tech giants of the "existential dangers" posed by artificial intelligence falling into the wrong hands during a conference in Scotland on Thursday.

The gathering at Dumfries House brought together senior representatives from Nvidia, Google DeepMind, OpenAI, Anthropic, and Britain's minister for artificial intelligence.

The debate over the risks of the technology has intensified in recent months, fuelled by several incidents and apocalyptic warnings from industry professionals.

"There seems urgency in adequately considering the existential dangers of such technologies falling into the wrong hands, and being used in potentially catastrophic ways," Charles said at the opening of the conference.

"Those in our world who value our humanity and its vital moral component are anxiously seeking your reassurance that we will not lose control of our destiny."

The British monarch called on tech leaders to ensure technology "remains firmly in the service of humanity, community and the natural world".

Among the guests at the conference were DeepMind chair Demis Hassabis, the head of American semiconductor giant Nvidia Jensen Huang and OpenAI's chief financial officer Sarah Friar.

Paolo Benanti, the Vatican's AI adviser, and Israeli author and historian Yuval Noah Harari also attended.

Buckingham Palace expects the conference to mark the beginning of a process to draw up common principles guiding the development and use of AI.

The king, who did not take part in the discussions, urged participants to consider how to harness AI "by putting safety at its heart" and to build international cooperation so that "no nation is left behind".

The conference coincided with dramatic claims by the late Princess Diana's brother that Charles acted insensitively following her death in a car crash almost 30 years ago.

Buckingham Palace hit back after Charles Spencer claimed in his new memoir that the then-prince, Diana's former husband, said "we'll forget her soon enough" days after she died in 1997.

"His Majesty is mindful that the pain of fraternal grief can cloud reason, affect judgement and colour memory in ways others do not recognise, even many years after such a loss," a palace spokesperson said.

Rapid advances in AI have fuelled concerns over job losses as well as the soaring energy demand and the environmental impact of data centres.

Fears that designers of AI agents could lose control of their creations mounted after several models from OpenAI and Anthropic reportedly broke out of their confined environments on their own, accessed the internet, and intruded on websites and platforms.

"The chance of something bad accidentally happening, if we get things wrong, is definitely non-zero," DeepMind's Hassabis said in his opening remarks at the summit.

"We've already seen the beginnings of this with the recent cyber incidents involving rogue AI agents, and there could be other, much more serious risks on the horizon, including biological and nuclear threats," he warned.

Nvidia's Jensen Huang, meanwhile, emphasised the benefits of AI in research and medicine, as well as its potential for "creating jobs in entirely new industries".

He told reporters that "we have got to debunk some of the myths out there," and called for "responsible optimism".

James Arroyo, the head of the Ditchley Foundation, a UK non-profit that facilitated the sessions, said: "We hope we can build on the really useful exchanges we had today" after the summit ended.

Anthropic CEO Dario Amodei had set off alarm bells on Saturday when he called on companies in the sector to slow down their development in order to better understand the risks, receiving support from OpenAI CEO Sam Altman and Elon Musk.

US President Donald Trump and several members of his government have, by contrast, reiterated in recent days their determination not to constrain the sector, out of concern that doing so would allow China to gain the upper hand.

Mark Zuckerberg, the head of Meta, has also rejected calls to slow the development, arguing that market forces and the risk of litigation are the best safeguards.