Summary

I tested whether supplying architectural guidance before an agent generates code changes the implementation it chooses. Round six improved governed compliance from 8/15 to 12/15 and cut scope expansions from 5/15 to 2/15, but functional completion fell by two and the run failed its locked gate. A blinded diagnosis traced both failures to two storage trials with an ambiguous fixture, not to the guidance mechanism. An isolated two-by-two confirmed it: with an explicit target, role-aware guidance was 3/3 compliant against baseline’s 0/3, with both at 3/3 completion. The implementation was retained, the experiment was closed, and no production-effectiveness claim follows.

Almost every control we have built for AI-generated code sits downstream of the thing it is trying to control. Code review, tests, static analysis, CI, human approval. They share a property: by the time they run, the architectural decision has already been made. Review can reject it, but rejecting it means throwing away working code and paying for the work twice.

I wanted to know whether intervening before generation changes what an agent builds, rather than detecting afterwards what it built. So I ran the experiment. It did not go cleanly, and that turned out to be the interesting part.

The hypothesis, stated so it could fail

If an agent receives task-relevant architectural guidance immediately before implementation, architectural compliance should increase without materially reducing its ability to complete the task.

The independent variable was baseline versus role-aware pre-generation guidance. The measured outputs were architectural compliance, functional completion, and scope expansion. Deliberately not does this product improve that agent, which is both narrower and less defensible than the mechanism question.

The defect I was actually testing

The original problem was not that the agent ignored decisions. It was what I came to call action-shape ambiguity: every retrieved decision was rendered as equally actionable. Give an agent an authentication task, retrieve an adjacent storage decision alongside it, and the storage decision could become an unrequested SQLite implementation. The agent was not disobeying. It was reading a constraint as an instruction.

The change assigns the unique top-scoring decision a direct role, marks the remaining selected decisions as adjacent_constraint, and renders the two differently. Retrieval, ranking, K and enforcement are untouched. If two decisions tie for top score, no direct anchor is produced at all, which is the conservative failure mode.

The first result was uncomfortable

Round six preserved a +4 governed-compliance count: 12 of 15 in treatment against 8 of 15 at baseline. Governed scope expansions fell from 5 of 15 to 2 of 15. Both of those point the right way.

It still failed. Functional completion in treatment was −2, and treatment retained two scope expansions, so the run missed its locked product mechanism gate. Under the frozen rule, R6 is permanently a FAIL. Not a marginal pass, not a pass with caveats.

Guidance that makes an agent more compliant and less able to finish the job has not obviously improved anything.

If I had stopped there, the honest conclusion would have been that the mechanism looked promising and was not validated. I want that sentence on the record, because what came next could otherwise read as motivated reasoning.

Debugging the experiment instead of explaining away the result

The failures were not evenly distributed, which is usually a sign that the headline number is hiding something. I ran a blinded post-R6 diagnosis before forming a view about the cause.

It found that no adjacent authentication constraint became an implementation objective. The leakage mechanism I had built the fix for was not what failed. Both failed gates traced to two storage trials that created parallel subclasses and left the supplied class incomplete: one localized direct-guidance implementation-targeting failure, not a general degradation.

So I isolated the storage task with a locked two-by-two, varying guidance against target explicitness. It classified the result as fixture_ambiguity_dominates. With an explicit instruction to implement the existing SessionStore, role-aware guidance was 3 of 3 architecturally compliant, 3 of 3 functionally complete, and 0 of 3 on scope expansion. Baseline was also 3 of 3 complete and 0 of 3 on expansion, but 0 of 3 compliant.

The completion problem was the fixture being ambiguous about what completion meant. The compliance difference survived removing it.

What this establishes, in three tiers

Supported. The role-aware presentation fixed the diagnosed adjacent-decision leakage and preserved a strong architectural-compliance signal in synthetic mechanism tests. That is why the implementation was retained.

Not supported. Production effectiveness. A precise estimate of ordinary model variance. Any claim that R6 passed. The two-by-two had three trials per cell, which is enough to locate a cause and nowhere near enough to size an effect.

Closed. Synthetic experimentation stops here. There is no R7 and there will not be one. The production-effectiveness A/B remains paused, and replacing its failed prerequisite would need a fresh prospective design rather than a reinterpretation of a run that already failed.

What the failure actually taught me

Four things, and only one of them is about the product.

A failing gate is more informative than a passing one. Had R6 cleared its threshold I would have shipped a conclusion built on a fixture I had never examined. The failure forced an inspection that found a real defect in my test design. The result I wanted would have hidden it.

Compliance and completion are separate axes, and moving one can move the other. I designed the gate to require both, which is why the run failed. That was the right call. A mechanism that improves architectural conformance while degrading delivery is not a win, and any measurement framework that reports only the flattering axis will eventually mislead the person running it.

Ambiguity in the task is a confound that looks exactly like a regression. Two storage trials produced parallel subclasses and left the supplied class incomplete. On the summary that reads as guidance harming completion. In the isolated two-by-two it disappears the moment the target is stated explicitly. If I had not blinded the diagnosis before forming a hypothesis, I would very likely have found the story I preferred.

Diagnose before you re-run. The instinct after a failed round is to change something and go again. Re-running would have produced another number and no understanding. The blinded diagnosis and the isolation study cost less than a fresh campaign and answered the question the campaign could not.

What happens next

The implementation stays. It fixed the leakage mechanism it was built for, and that finding survived the failed gate.

The synthetic programme stops. There is no R7. Running another confirmatory round against fixtures I control would tell me about my fixtures, not about software engineering, and at three trials per cell the marginal information is close to zero.

What would actually move this forward is a different shape of study:

The production-effectiveness A/B remains paused, and it stays paused. Its prerequisite failed, and replacing a failed prerequisite requires a fresh prospective design, not a reinterpretation of the run that failed.

Why publish this at all

The temptation to re-run until something passes is real, and it is exactly what makes most vendor benchmarks worthless. Publishing only the rounds that clear the bar produces a literature where every mechanism works and nothing is learned.

I am more interested in finding where this mechanism stops working than in producing another synthetic result that clears a threshold I chose myself. That is a slower way to build a product and a faster way to find out whether the thing you built is real.