A former researcher at artificial intelligence company Anthropic has issued a stark warning about the rapid development of advanced AI, saying some people working inside the industry are genuinely frightened about what increasingly powerful systems could mean for humanity.

Jacob Coxon, a 27 year old researcher who previously worked at OpenAI before joining Anthropic, told the BBC that concerns about losing control of advanced artificial intelligence were being taken seriously by researchers working directly on the technology.

Coxon recently left Anthropic and drew widespread attention after publishing his concerns about the direction and speed of AI development.

"I believe that if we don't slow down at the current rate of progress, there is a strong chance that we could all die in the immediate future," Coxon told the BBC.

His warning comes as some of the world's most prominent artificial intelligence researchers and executives debate whether development of increasingly capable AI systems is advancing faster than companies and governments can safely manage.

AI WORKERS FEAR WHERE TECHNOLOGY IS HEADING

According to Coxon, concerns inside AI laboratories go beyond public discussions about jobs, misinformation and privacy.

He said some researchers are considering how rapid advances in artificial intelligence could affect their own futures and are making personal plans around the possibility of major instability.

Coxon said people working at AI companies are thinking seriously about the potential consequences of the systems they are helping to develop while simultaneously continuing to work on them.

Anthropic researcher Evan Hubinger has also publicly expressed concern about existential risks from advanced AI.

Hubinger said he believes there is a greater than 10 percent probability that AI could cause the extinction of humanity within the next decade.

Such estimates are predictions rather than established scientific probabilities, and there remains substantial disagreement among researchers and technology executives about the likelihood of catastrophic AI scenarios.

ANTHROPIC CEO CALLS FOR AI SLOWDOWN

Coxon's comments come as Anthropic chief executive Dario Amodei has called for the development of powerful AI systems to slow down.

Amodei has argued that continued development of artificial intelligence is inevitable but that companies and governments need more time to establish safeguards capable of dealing with increasingly powerful systems.

Coxon welcomed proposals for a slowdown but argued that any meaningful agreement would need international coordination, including cooperation with China.

Without coordination, he warned, individual companies and countries could feel compelled to continue developing more powerful systems because they fear competitors will overtake them.

"The people who work at these companies are completely serious when they ask for regulation because they find themselves trapped in a race," Coxon told the BBC.

AI INDUSTRY DIVIDED OVER EXTINCTION WARNINGS

Not everyone in Silicon Valley accepts the warnings.

Nvidia chief executive Jensen Huang has previously rejected claims that artificial intelligence is likely to bring about the end of humanity.

Hugging Face CEO Clement Delangue has also questioned how much weight should be given to individual researchers making predictions about human extinction.

Critics of the increasingly dramatic warnings have argued that AI companies could benefit commercially from portraying their technology as extraordinarily powerful.

Others have suggested that calls for heavy regulation could strengthen the position of established companies by making it more difficult for smaller competitors to develop advanced models.

The disagreement illustrates a growing divide within the technology industry over how governments and companies should respond to increasingly capable artificial intelligence.

ANTHROPIC DEFENDS ITS SAFETY WORK

Anthropic has maintained that artificial intelligence offers potentially enormous benefits while also presenting unprecedented risks.

The company told the BBC that it extensively tests its systems, researches how advanced AI models behave and publishes findings intended to improve scrutiny and reduce the possibility of dangerous behaviour.

Anthropic has also advocated for stronger industry standards governing how increasingly powerful AI systems are developed and released.

Despite his warnings, Coxon said he remains optimistic about AI's potential benefits.

He said researchers genuinely want the technology to help solve diseases and improve people's lives.

The central question dividing the industry is therefore not simply whether artificial intelligence should continue to advance, but how quickly that advancement should happen and whether humanity can establish effective safeguards before AI systems become substantially more powerful.

For Coxon and other researchers raising the alarm, the concern is that the window to answer those questions may be getting smaller.