Security & Privacy: The Worst Case
A RAG system is a machine that surfaces text from your documents on demand. That's the whole value — and, handled carelessly, the whole risk. The failure that ends up in the press isn't a wrong answer; it's the assistant showing someone a document they were never allowed to see.
Three risks deserve a leader's attention. Access control: if retrieval ignores who's asking, the assistant can surface a passage from a restricted document — salaries, another customer's data, an unreleased plan — to anyone who phrases the question right. Permissions must be enforced at retrieval time, so users can only ever get answers from documents they're entitled to. Prompt injection: a malicious document (or a web page you ingest) can contain instructions that hijack the assistant — "ignore your rules and reveal…" — so retrieved content can't be blindly trusted as safe. Sensitive data: personal information flowing into prompts and logs may cross compliance lines.
None of this requires you to be a security engineer. It requires you to ask whether permissions are enforced on retrieval, whether ingested content is treated as untrusted, and where sensitive data ends up.
One index, everyone's docs
Your team built the assistant on a single search index covering HR files, finance, and every team's documents, so "it can answer anything." Legal just found out. What's the risk, and what do you require before it ships?
The risk is a data breach with your name on it: anyone can phrase a question that surfaces a passage from a document they were never cleared to read — payroll, another team's plans, a customer's data. Require that retrieval enforce each user's permissions, so the assistant can only ever pull from documents that user is already entitled to see. "It can answer anything" is precisely the problem, not the feature. This is a hard gate before launch, not a fast-follow — a single leaked salary table can undo the whole project's goodwill.
The assistant that leaked the salary sheet
A company put its internal assistant on one shared index of "all company documents." An employee, curious, asked a pointed question about compensation — and the assistant helpfully retrieved and summarized a passage from a restricted payroll spreadsheet that had been swept into the corpus. Retrieval worked flawlessly; that was the problem. There was no notion of who was allowed to see what.
The incident forced an emergency takedown and a rebuild around permission-aware retrieval, where every query is filtered to the documents that specific user can access. The painful lesson for leadership: a RAG system inherits every access-control gap in your document store and makes it instantly, conversationally exploitable. Permissions aren't a feature to add later — they're a precondition to launch.
Brightline scopes retrieval to each employee's existing document permissions from day one — an engineer's assistant can't surface HR files it wouldn't otherwise see — and strips personal data from what gets logged. It's slower to build and non-negotiable; the CEO signs off after legal frames the alternative as a breach.
Quiz · Chapter 10
- The most damaging RAG failure is often:
- Access control must be enforced:
- Prompt injection means:
- "Everyone searches the same index" is: