Pre-LLM infrastructure failures
Six recurring failure classes appeared across evaluated human-facing AI deployments. Each one involves infrastructure or application behavior that a system prompt does not independently control.
Data leaves before redaction
Sensitive input reaches the network before the application applies a privacy control.
In evaluated deployments, names, dates of birth, government identifiers, insurance information, and health details were visible in raw request bodies sent to third-party infrastructure. Once transmission has occurred, a later application-layer redaction cannot undo it. The control must operate before the outbound model request.
Crisis signals follow the normal route
A consequential safety signal is processed as an ordinary conversational turn.
Evaluated paths sent crisis or suicide-adjacent language through the standard model flow and returned generative empathy with a resource appended later. Network inspection did not show a separate deterministic intercept or routing event before the model call. A helpful-sounding response is different from a verified safety workflow.
A handoff is described but never executed
The response claims an escalation, while the corresponding action record is absent.
In one evaluated flow, the assistant told the user it was initiating a human handoff while the associated function log remained empty. The user-facing sentence described an action, but the application record showed no action occurred. Reliable governance must verify execution separately from generated language.
Data requests receive only reassurance
A conversational acknowledgment is not matched by a verified data-handling event.
After explicit deletion or opt-out requests, evaluated chatbots provided reassuring language while session identifiers and user-linked state remained active in subsequent payloads. No purge event or context reset was visible. A data-rights response should connect to an executed operation and an inspectable record.
Internal information is exposed
An insufficiently protected endpoint returns prompts, configuration, or internal diagnostic data.
Standard browser inspection identified public responses containing system instructions, restricted URLs, internal rules, knowledge-source identifiers, and diagnostic or reasoning-like fields. These observations point to access-control and response-shaping failures around the model, rather than a weakness that prompt wording alone can repair.
The conversation exceeds its intended role
A support or administrative assistant moves into professional guidance beyond its intended scope.
Evaluated assistants positioned for administrative, coaching, wellness, or support work moved into emotional counseling, behavioral-health guidance, legal advice, or eligibility guidance when users presented related questions. Role instructions did not consistently hold the boundary. Consequential categories need explicit routing, escalation, and output controls.
Evaluation method
Compare the conversation with the system record.
The evaluation series used the Cooperative Extraction Method alongside browser network inspection. Reviewers compared what the assistant said with the request payload, response fields, function or action logs, and observable state after the interaction.
- Before the model: Inspect what data was transmitted and whether a deterministic intercept ran first.
- During the workflow: Look for routing fields, function calls, holds, or escalation events corresponding to the assistant’s claims.
- After the response: Check whether identifiers, session state, and application records changed as represented.
The source inventory records responsible disclosure before publication. These findings describe the deployments and configurations that were evaluated; they are not claims about every AI application or legal conclusions about a particular operator.