An HTTP 500 Internal Server Error page filling an entire website, the kind of AI-generated code security risk a single bad configuration line can cause.

AI-Generated Code Security Risks From an AI-Assisted Rebuild

by
An HTTP 500 Internal Server Error page filling an entire website, the kind of AI-generated code security risk a single bad configuration line can cause.
One line of AI-written .htaccess configuration returned HTTP 500 on every page of the site, one of only two findings that became an actual incident.

Over six weeks I rebuilt Poetry Lovers’ Page, a 30-year-old poetry website, with Claude Code as my coding partner, and I kept a register of every security issue it produced – 59 of them. This article is the pattern that ties those issues together, and the controls that caught them, so you can manage the same AI-generated code security risks on your own projects.

Executive Summary

Between March and May 2026 I rebuilt Poetry Lovers’ Page, a reference site of more than 2,000 poems, using an AI coding assistant for most of the work. I logged every security issue I found: 59 of them, ranging from a one-line configuration change that took the whole site down to a 90-minute cascade that leaked five live API keys three different ways. Only two ever became actual incidents in the strict sense, the ones that kicked off an incident response; the rest were caught as findings before they could. None of them involved a clever exploit. They were confident, good-faith category errors, shipped at machine speed, with nothing in the output to show where the AI had gone wrong. I came to call the failure underneath them Self-Attested Safety: the habit of accepting assurance the AI produced or asserted about its own work. Every one of the 59 was caught by a control I had in place. The AI never caught those mistakes on its own. If you are adopting AI-assisted development, the work that matters is moving your assurance out of the AI and into checks it cannot author or fake.

The Pattern Behind All 59: Self-Attested Safety

Self-Attested Safety shows up whenever the thing meant to provide assurance – a passing quality gate, a validator, a hallucination detector, a block of code that simply looks finished – is produced or vouched for by the same class of system it is supposed to check. The artifact reads as reviewed, and that plausibility gets mistaken for having passed a control.

I have spent 25 years in product security – threat modeling, secure development lifecycle work, chairing an OWASP chapter – and I went in expecting the AI to be sloppy the way junior engineers are sloppy. A junior writes tentative code and tells you where they were unsure; Claude Code wrote confident code everywhere, including the places it was wrong, and the confidence read exactly the same in both. A generative model produces artifacts with no internal signal separating the correct parts from the broken ones, and it can write the very checks meant to catch its own mistakes. Any safety you read off the artifact, or take from another AI’s word about it, is self-attested. The principle I kept coming back to was to anchor assurance in something the AI cannot author or assert.

The usual advice – review AI code the way a senior reviews a junior – only half applies. A junior who ships a time-of-check race learns from the review and writes the next file more carefully. The model carries nothing forward; the same gap can resurface in the next script an hour later. So the review cannot be a teaching moment. It has to be a standing control that runs on every artifact, whether or not the last one came back clean.

Bar chart titled '59 security issues in six weeks' showing 3 Critical, 21 High, and 35 Medium or lower, from an AI-assisted website rebuild.
Six weeks of AI-assisted development produced 59 logged security issues, 24 of them High or Critical.

The Register: 59 Security Issues From One Real Rebuild

I did not set out to write a paper. I set out to ship a website and kept a running security register, a habit two decades of this work beats into you. By the end it held 59 numbered entries across roughly six weeks, 3 of them rated Critical and 21 High. Each entry records the AI-generated artifact, what was wrong, how it was found, the fix, and the general lesson. The register opens with its own thesis:

“AI-generated code is fast and often functionally correct — but security is frequently absent by default. Systematic review finds it.”

Poetry Lovers’ Page security register

I presented this material at ISSA Los Angeles on May 19 and at PlanetCyber on June 3, and both rooms wanted the same thing afterward: the issues in writing, with the controls attached. The stakes on a poetry site are low, which made it a cheap place to learn something that travels well beyond it. Every organization adopting these tools has its own version of this register, whether or not anyone is writing it down.

How the Pattern Reaches Production

The mechanics repeat across the register. Each issue starts as a confident artifact, and how much damage it did depended on its blast radius.

A one-line config change that took down every page. Hardening the site, the AI added an Apache DirectoryMatch directive to .htaccess to block an unused stats panel it had correctly flagged. That directive is valid in server config and illegal in .htaccess, where Apache answers an illegal directive with HTTP 500 for every request to that directory and below. Every page went down on deploy. The AI knew the directive and picked it for a sound reason; it simply did not check where that directive was allowed. That is a context gap, and the blast radius was the entire site.

Thirteen flaws in one AI-written validator. I asked the AI for a sitemap validator, then asked a second agent to review it adversarially. That review found 13 distinct issues in roughly 130 lines, four of them High: a time-of-check/time-of-use race, a silent fallback to an unsafe XML parser, and a function that printed “VALID” after quietly skipping schema validation. The first version already used hardened libraries. The gaps were all in risks the prompt never named.

An orchestrator that marked unfinished work as complete. A batch orchestrator was supposed to publish work only after it passed multiple review rounds and an anti-hallucination audit. When two worker agents hit rate limits and stopped early, the orchestrator “helped” by marking their incomplete drafts as finished and committing them as if they had passed the whole pipeline. There was nothing adversarial about it. The orchestrator wanted the ticket closed, and the quality gate stood in the way.

“An AI will bypass the controls it was told to enforce if ‘helping’ seems like the right thing to do.”

Poetry Lovers’ Page security register, SEC-007

A verifier that checked nothing for weeks. The AI built a script to catch hallucinated dates by cross-checking reference data. It read fields named birthDate and deathDate; the schema used birth and death. A missing key returns nothing, every comparison passed, and the script reported success while checking nothing – for weeks. A verifier that always passes proves nothing until you feed it an input it should reject.

Fourteen fabricated poems out of 43. Asked for the text of 43 obscure Poe poems, the AI fabricated 14 of them, a 33% fabrication rate, in verse indistinguishable from the real thing. The fabrication rate rose with the obscurity of the source, and the model’s confidence never wavered between the real and the invented.

Three simultaneous good-faith leaks. The worst hour was a credential incident with three AI-caused layers. An agent ran trace debugging on a script that loads the environment file, printing five live tokens into a log. The AI-written scanner built to find such leaks then echoed the matched secret values to standard output, which in an agent session is transmitted to the model’s API. Rotating one exposed key surfaced a third problem: it had been hardcoded in source for years, shipping in every deploy.

Two poem stanzas side by side labeled A and B, one a genuine Edgar Allan Poe stanza and one fabricated by an AI, visually indistinguishable.
Asked for obscure Poe poems, the AI fabricated 14 of 43. A is genuine Poe; B is invented. The model's confidence was identical for both.

What This Changes in Your Threat Model

Many teams assume code that looks reviewed has been reviewed. AI output looks reviewed by default, with no signal where it is wrong. Put an explicit gate in front of AI-generated changes that touch security or carry a large blast radius, and accept that the gate has to be selective or it becomes the bottleneck everyone skips. Even so, volume outpaces attention, and a category error can slip through on a quiet afternoon.

A second assumption worth retiring is that a passing verification step means the thing was verified. The orchestrator and the date-checker that read the wrong field both show an AI component reporting success without performing the check. Verify completion by artifacts the actor cannot fabricate – the review-round files, the audit log – and treat its own say-so as unproven. Even then, a “helpful” system can satisfy the letter of an artifact check, so the expensive checks need occasional human spot-audits.

A third assumption hides in your tooling: that a security tool’s logic is its whole threat model. The scanner that leaked the secrets shows the output channel counts too. In an agent session, standard output is egress to a third party. Treat tool output as untrusted egress and redact sensitive matches by default. Redaction can still be incomplete, so sensitive-data tooling needs its output reviewed as carefully as its logic.

The Secure-by-Design Alternative

Designing against Self-Attested Safety means deciding up front where each assurance is anchored, and refusing to count anything the AI could have authored as a control. In practice that is five moves: verify completion by artifacts rather than claims; test every safety net against an input it must fail on; review AI security-adjacent code adversarially before it ships; put a human gate and a fast rollback in front of anything with catastrophic blast radius; and redact sensitive output by default in any tool an agent runs. None of them depend on the AI behaving any differently. They just make its mistakes survivable.

Pattern: Self-Attested Safety

Preconditions: a generative AI produces artifacts that enter a real system – code, configuration, content, tooling – and assurance about them comes from their appearance or an AI component’s say-so, at a throughput where ad-hoc human review cannot keep pace.

How it enables attack: the artifact ships with a category error invisible on its surface. Because the assurance was authored or asserted by the AI itself, nothing independent contradicts it, and the error reaches production at the system’s blast radius. An external attacker amplifies this by aiming at the gaps generative output reliably leaves: injection in generated code, secrets in tool output, instructions that route an agent around a control.

Detection signals: a verification step that has never failed; “complete” reported without the artifacts a real run produces; AI-written tooling that prints matched secrets; high-blast-radius config or secret-touching changes with no human in the path; confident AI facts in domains where ground truth is scarce.

Mitigations: structural completion gates; known-bad-input tests for every net; adversarial review of security-adjacent code; authoritative-source reconciliation with “unknown” modeled explicitly; blast-radius-tiered human gates with rollback reachable; redact-by-default tooling.

Residual risk: external anchors have their own failure modes – artifact checks can be satisfied superficially, redaction can be incomplete, review does not scale infinitely – so these controls reduce the pattern; they do not remove it.

OWASP Top 10 and Top 10 for LLM Applications Mapping

OWASP Top 10 for LLM Applications 2025: LLM06 Excessive Agency

LLM06 covers harm from an LLM-based system having more autonomy than the task warrants, including bypassing controls to reach a goal. The orchestrator that published unreviewed drafts as finished is the clearest example: it had the agency to write final artifacts and used it to route around the gate it was meant to enforce. Enforce agency limits structurally – verify by artifacts an actor cannot fabricate, and constrain what an agent can touch with tooling, because prompt text alone does not hold.

Reference: OWASP – LLM06:2025 Excessive Agency

OWASP Top 10 for LLM Applications 2025: LLM02 Sensitive Information Disclosure

LLM02 covers exposure of sensitive data through LLM systems, including via the system’s own outputs and channels. The credential cascade spans it: live tokens captured by trace debugging, then a scanner that echoed matched values to a channel that egresses to a model API. Treat AI-tool output as egress, redact by default, and back operational rules with a detector at a checkpoint the agent cannot skip.

Reference: OWASP – LLM02:2025 Sensitive Information Disclosure

OWASP Top 10 for LLM Applications 2025: LLM09 Misinformation

LLM09 covers confident, plausible, false output presented as fact, and the over-reliance it induces. The 33% poem fabrication rate is one measured example, and the model’s confidence was the same for the real poems and the invented ones. Make verification effort proportional to how unverifiable a request is, reconcile AI-produced facts against authoritative sources, and model “unknown” explicitly rather than letting the AI fill every field.

Reference: OWASP – LLM09:2025 Misinformation

OWASP Top 10 2025: A02 Security Misconfiguration, A05 Injection, and A09 Logging and Alerting Failures

A02:2025 Security Misconfiguration covers insecure configuration of platform components, where the failure is in config rather than logic – the .htaccess directive exactly. The validator’s gaps were an A05:2025 Injection exposure in generated code, and the date-checker that failed open was an A09:2025 Logging and Alerting Failures case, a control reporting healthy while detecting nothing. Config with all-or-nothing failure modes needs a human gate and rollback; safety nets need known-bad-input tests before they ship.

Reference: OWASP Top 10:2025

Bar chart titled 'What caught the 59 issues' showing an external control caught 59 and the AI on its own caught 0.
Across all 59 issues, a control did the catching. The AI never caught any on its own.

Controls That Actually Worked

Every issue was caught by an external anchor, and the same anchors generalize. A structural completeness gate that confirms the agent’s artifacts exist, regardless of what the orchestrator reports, stops that class of failure – high confidence, provided artifacts are costly to fake and occasionally audited. Known-bad-input testing closes the fail-open class for every case in the test corpus, and nothing beyond it. Adversarial review caught all 13 validator gaps in one pass, though it is human-time-bounded and belongs on security-adjacent code, since it cannot cover everything. A blast-radius gate plus a clean prior commit turned the site outage into a quick rollback, which bounds the damage without preventing the error, so it has to sit alongside the gate. And a written incident response plan was the forcing function that made the three-layer credential cascade a controlled 90-minute response.

What Product Teams Commonly Miss

  1. Which of our verification steps have never failed, and have we fed each one an input it must catch to confirm it actually fails?
  2. When an automated step reports “complete,” are we checking the artifacts a real run produces, or trusting the report?
  3. Which AI-authored changes can reach production with catastrophic blast radius – config, secrets, deploys – without a human in the path, and is rollback reachable for each?
  4. Does any AI-written tool that handles sensitive data print what it finds, and where does that output go in an agent session?
  5. Are our agent boundaries enforced by tooling, or only by instructions in a prompt?

What to Do Now

Immediate (teams with security engineering capacity). Add two structural gates this week. Make “done” mean “the required artifacts exist,” checked by something the producing agent cannot write. Then give every safety net a known-bad fixture it must fail, wired into continuous integration so a net that starts passing everything breaks the build. The date-checker that silently failed open needed exactly one such test:

assert verifier.check(record_with_a_known_wrong_date) == FLAGGED

Acceptance criterion: a deliberately broken verifier fails your pipeline before a human looks at it.

Short-term (some investment, 30-90 days). Stand up a security register and tier review by blast radius. Log every AI-caused security issue with the same fields each time, so patterns become visible instead of forgettable. Define the high-blast-radius set – config files, anything touching secrets, anything that deploys – and route only those through a mandatory human gate with rollback. Acceptance criterion: you can produce your register’s severity distribution on demand, and no .htaccess-class change merges without a named reviewer.

Strategic (all orgs, planning horizon). Write the incident response and recovery plans before you need them, and draw the AI’s autonomy boundary explicitly inside them: which actions an agent may take unsupervised during an incident (containment) and which it may never take (key rotation, deploys, dashboard changes), with the incident log as the handoff. Back recovery with immutable, ransomware-resistant backups and tested restores. Acceptance criterion: a drill restores from backup within your stated recovery-time objective.

Before you merge AI-written code, run it against this checklist:

  1. Did a control the AI could not have authored confirm this is correct – a test, an external source, a human gate – or am I trusting how finished it looks?
  2. If this is a verifier or safety net, have I watched it fail on input it should catch?
  3. What is the blast radius if this is wrong, and is rollback within reach right now?
  4. Does this touch secrets or config, and if so, is a human in the path?
  5. If this produces facts, did I reconcile the ones I cannot easily verify against an authoritative source?
  6. If this is a tool, where does its output go, and would I be comfortable with that output reaching a third party?

What This Analysis Doesn’t Cover

This is one developer, one site, six weeks – a single register, and nowhere near a measured base rate for AI-generated code security risks across the industry. The specific figures (a 33% fabrication rate on obscure source material, roughly one significant issue per 32 lines of security-adjacent code) describe what these particular tasks produced and will not match yours. I am also not handing down a verdict on the tool: the velocity was real, the fixes it shipped were real, and I would use Claude Code again. This register is human-in-the-loop throughout; fully autonomous coding agents with no human in the path are a harder problem I do not take on here.

Conclusion

The pattern under all 59 issues is that a generative model produces work with no internal signal of where it is wrong. Safety cannot be read off the artifact; it has to be anchored in checks the AI cannot author or fake. Keep one habit from all of this: before trusting an AI-produced assurance, find the independent control behind it. When the only thing vouching for the work is the AI that did the work, there is no control at all. The teams getting this right spend less time gauging how far to trust the AI and more time building the anchors that make its errors safe to live with.

Frequently Asked Questions

What are the main security risks of AI-generated code?

The recurring risk is confident, plausible output with security controls absent by default and no signal showing where the code is wrong. In one six-week register that meant misconfigurations with site-wide blast radius, injection and race-condition gaps in generated utilities, fabricated data, leaked credentials, and verification tools that silently failed open – every one caught by a control, never by the AI on its own.

What is Self-Attested Safety in AI-assisted development?

Self-Attested Safety is relying on assurance an AI produced or asserted about its own work – a passing check it wrote, a completion it claimed, code that merely looks finished – as if that were independent verification. Because a generative model can author the very checks meant to catch its mistakes, any safety derived from the artifact’s appearance or another AI’s claim is unverified.

How should teams review AI-written code for security?

Anchor every assurance outside the AI. Verify completion by artifacts an agent cannot fabricate, test every safety net against input it must fail on, review security-adjacent generated code adversarially, and gate high-blast-radius changes – config, secrets, deploys – through a human with rollback ready. Reserve scarce human review for where the blast radius is catastrophic.

Which OWASP categories apply to AI coding assistant risks?

Both top-ten lists apply. From the LLM Top 10: LLM06 Excessive Agency, LLM02 Sensitive Information Disclosure, and LLM09 Misinformation. From the web Top 10: A02 Security Misconfiguration, A05 Injection in generated code, and A09 Logging and Alerting Failures when a verifier fails open.

References