An investigation involving nearly 2 million AI prompts found evidence of major privacy and security failures across the inference supply chain. At least three AI providers were compromised, multiple resellers in the Inference Marketplace recorded and shared prompts, and some providers failed to honor privacy settings such as zero data retention (ZDR) and “no data collection.”
“We sent nearly 2 million AI prompts containing unique trackable secrets to AI providers and observed where those secrets surfaced,” said Moore, chief research officer at security firm Dragos. Moore wrote on Thursday. “What we discovered revealed security breaches, rapid theft, and widespread failures to protect customer data.”
AI prompts are encrypted while in transit, but they are decrypted at every hop in the large network of inference resellers. A prompt is initially sent to an AI router, which reads its plain text to select an inference provider. Requests may be submitted to a pool of models running on different hosting providers, or passed from one router, such as Vercel, to another, such as OpenRouter. Each intermediary may log and retain prompts for the period required by its policies—or indefinitely.
A “Wild West” in the AI era
The most serious finding involved the compromise of three separate hosting providers. After gaining access to one provider, attackers reused “hundreds” of AWS keys, Moore wrote:
Each key was sent in one request to one provider, all within a short period of time in late October, and all requests sent during that period were stolen. Some of the stolen keys came from requests that were rejected by the provider before the model was run. This indicates that the attacker was copying requests at the provider’s front door rather than deeper in the inference stack. In the other two breaches, some of the stolen requests were marked as ZDR. Privacy flags are useless if the attacker is already on the inside.
The attackers tested each compromised key—sometimes within hours of its introduction—to determine which resources it could access. They searched S3 buckets, secret managers, and parameter stores for additional credentials. The investigation also found an instance in which a model hosted on AWS Bedrock was enabled and an IAM user was added to preserve access. In total, the stolen decoy keys were used in “tens of thousands” of AWS API calls.
Source: arstechnica.com












