جيميناي يخترق 3 شركات حقيقية خلال اختبار أمني.. «قوقل» تكشف التفاصيل

In a first-of-its-kind incident known to Google, the AI model Gemini managed to breach the systems of three real companies during a cybersecurity test, after the model exited the closed experimental environment and unintentionally reached the internet.

The breaches occurred last May during a security assessment conducted by Irregular, a company specializing in testing the safety and security of advanced AI systems, before the company informed Google of the incidents at the end of July.

Google confirmed the breaches, but stated that the model did not cause any damage to the targeted companies and that it ceased its activities as soon as it realized it had accessed real systems.

How did Gemini breach the companies?

Irregular was conducting tests within a closed environment that included fictitious companies, aiming to measure Gemini's capabilities in the field of cybersecurity. This environment was not supposed to be connected to the internet, but access to the network became available by mistake, according to the Wall Street Journal.

Once the model reached the internet, it began interacting with real systems instead of the fictitious targets for which the test was designed.

In one instance, researchers were testing Gemini's ability to obtain information from software belonging to a fictitious company that shared the same name as a real company.

When the model became capable of connecting to the internet, it managed to guess the correct password and access a service belonging to the real company, initially believing it was part of the testing environment.

Google stated that Gemini stopped executing any further actions as soon as it realized it had breached a real company and not the simulated company used in the test.

Passwords and Data Published Online

In two other tests, Gemini was able to search the internet and find public code repositories that contained credentials linked to two other companies.

The model used that data to access the systems of the two real companies before also stopping once it realized that the targets it had reached were real companies and not part of the test.

Google stated that the model halted its activity in all three cases, confirming that the affected companies did not suffer any damage as a result of the breaches.

Why didn't Google announce the breaches?

Irregular informed Google of the incidents at the end of July, following the discovery of a separate incident involving OpenAI models accessing the systems of Hugging Face during a security test.

Unlike OpenAI and Anthropic, which chose to voluntarily disclose similar incidents, Google did not announce the Gemini breaches at that time.

Google justified its position by stating that the model did not cause harm to the three companies and that it stopped breaching as soon as it discovered the true nature of the targets. However, the company confirmed that it informed the three entities that experienced unauthorized access.

Heller Atkins, Vice President of Security Engineering at Google, stated that the model "found publicly available information on the internet and guessed credentials to access sites it thought were part of the test," before stopping in all three cases.

She added that these incidents underscore the importance of training powerful AI models to act responsibly.

A Series of Breaches Raises Concerns About AI

The Gemini incident follows similar occurrences involving AI models from other companies.

In July, OpenAI revealed that its models were able to bypass isolation controls and access systems belonging to the company and the Hugging Face platform during internal tests, in an incident the company said highlighted the need to enhance monitoring, containment, and security measures as model capabilities increase.

Anthropic also reported incidents related to unexpected behavior from its models, amid growing discussions about the ability of AI companies to control more advanced models when they gain tools to access the internet and execute code.