Security researchers employed ChatGPT as a co-conspirator to plunder sensitive data from Gmail inboxes without alerting users. The vulnerability exploited has been closed by OpenAI but it’s a good example of the new risks inherent to agentic AI.
The heist, called Shadow Leak and published by security firm Radware this week, relied on a quirk in how AI agents work. AI Agents are assistants that can act on your behalf without constant oversight, meaning they can surf the web and click on links. AI companies laud them as a massive timesaver after users authorize their access to personal emails, calendars, work documents, etc.
Radware researchers exploited this helpfulness with a form of attack called a prompt injection, instructions that effectively get the agent to work for the attacker. The powerful tools are impossible to prevent without prior knowledge of a working exploit and hackers have already deployed them in creative ways including rigging peer review, executing scams, and controlling a smart home. Users are often entirely unaware something has gone wrong as instructions can be hidden in plain sight (to humans), for example as white text on a white background.
The double agent in this case was OpenAI’s Deep Research, an AI tool embedded within ChatGPT that launched earlier this year. Radware researchers planted a prompt injection in an email sent to a Gmail inbox the agent had access to. There it waited.
When the user next tries to use Deep Research, they would unwittingly spring the trap. The agent would encounter the hidden instructions, which tasked it with searching for HR emails and personal details and smuggling these out to the hackers. The victim is still none the wiser.
Getting an agent to go rogue — as well as managing to successfully get data out undetected, which companies can take steps to prevent — is no easy task and there was a lot of trial and error. “This process was a rollercoaster of failed attempts, frustrating roadblocks, and, finally, a breakthrough,” the researchers said.
Unlike most prompt injections, the researchers said Shadow Leak executed on OpenAI’s cloud infrastructure and leaked data directly from there. This makes it invisible to standard cyber defenses, they wrote.
Radware said the study was a proof-of-concept and warned that other apps connected to Deep Research — including Outlook, GitHub, Google Drive, and Dropbox — may be vulnerable to similar attacks. “The same technique can be applied to these additional connectors to exfiltrate highly sensitive business data such as contracts, meeting notes or customer records,” they said.
OpenAI has now plugged the vulnerability flagged by Radware in June, the researchers said.
