How to keep API keys and secrets out of LLM prompts and context
Stop credentials, tokens and connection strings from entering prompts, context windows and model provider traffic, and know what to do if one does.
This is for engineering managers and security leads whose teams build AI features and internal agents. Developers under deadline pressure paste API keys into system prompts, let agents read config files, or retrieve documents that happen to contain passwords. Once a secret is in the context window, it can appear in an answer, a log or a provider's request record. This post explains where secrets leak into prompts, how to keep them out by design, and how to respond when one gets in.
A developer builds an internal agent that calls the billing API. The quickest route is to put the API key in the system prompt and ask the model to include it in requests. It works. Months later, a user asks the agent to show its instructions for debugging, and the full key appears in a chat transcript that is shared in a team channel.
Other leaks are quieter. An agent with file access reads an environment file while answering a question. A knowledge base built from old wiki pages includes a page with database credentials, which retrieval then places into context for any user who asks a related question. None of this needs an attacker, only ordinary use of a system that treats context as private.
A B2B SaaS company in Bengaluru built a support assistant over its internal wiki and runbooks. During a pre-launch review, the security lead ran a secret scanner over the indexed content and found staging database passwords and an old payment sandbox key embedded in troubleshooting pages. The team removed them from the wiki, rotated every affected credential, moved the assistant's ticketing API key out of the system prompt into a server-side tool, and added a runtime check that masks key-like strings in responses. After launch, the assistant answered the same support questions, the wiki owners added scanning to their page review routine, and the security team had a tested rotation runbook ready.
Is it safe to put a key in a prompt if the model provider does not train on our data?
No. Training is only one risk. The key can still be echoed in answers, stored in your own logs, or retained in provider request records under their terms. Keep keys in tool code where the model never sees them.
What patterns should a secret scanner look for?
Common key prefixes for your cloud, payment and messaging providers, private key blocks, connection strings, bearer tokens and high-entropy strings near words like password or secret. Tune it to the systems you actually use to reduce noise.
Our agent needs a credential to work. How does it authenticate?
The tool function authenticates on the server using a credential fetched from your secrets manager at call time, ideally a short-lived, narrowly scoped token. The model only requests the action; it never handles the credential itself.
Who should be alerted when a secret is detected?
Route alerts to the security owner and the owner of the affected system, with the source location and a request ID. The system owner rotates the credential; security confirms usage logs show no misuse.
Want this fixed in your business?
Send the workflow. You get a scope and a fixed price within two working days — or a straight answer that it isn't a fit.