Reliability is not safety: Compiled context raised observed correctness and unsafe acceptance

One arm produced 61 malformed answers. The other produced none. The second arm was less safe.

The study ran one pinned local Qwen3-4B model against the same questions and schedule. Raw context produced valid primary answers in 75 of 118 conditions. Compiled context produced valid primary answers in all 118.

Correctness rose by 8.98 points. Review burden fell by 26.28 points. Unsafe acceptance rose by 16.75 points and crossed the preregistered +5-point harm threshold.

The overall outcome was negative.

Evidence boundary: this is one completed local Qwen3-4B A versus C study. It compares raw and compiled context under a frozen protocol. It does not compare model sizes, establish a scaling law, calibrate confidence, prove general determinism, or provide an unconditional benchmark.

Reliable structure is not safe judgment.

What changed between A and C

Arm A received raw context. Arm C received context built through the deterministic compiler described in Context should be a build artifact, not a prompt assembled at runtime.

The model, questions, bases, schedule, scoring rules, and run boundary stayed fixed. Context construction changed.

The completed attempt covered 14 independent bases, split evenly between Distill and LLMTraceFX. It scheduled 118 conditions and 332 observations, including the repeat subset. The result contained 271 valid observations, 61 malformed JSON observations, and zero adapter errors.

This was not a retry chosen after seeing model output. The v2 protocol was frozen before any v2 model output. The corrected runner landed in Distill PR #109 at commit 670a48a, with all 12 CI checks green.

The correctness test used equal weight per independent base. Arm A scored 28.35%. Arm C scored 37.34%. The exact two-sided base-clustered sign-flip test gave p=0.09375. That result is not conventionally significant. The p-value applies only to correctness.

Review burden fell from 57.21% to 30.93%. Unsafe acceptance rose from 26.77% to 43.53%. No p-value was computed for unsafe acceptance. The preregistered decision rule did not need one: a rise above 5 points was harm. The observed rise was 16.75 points.

The malformed answers were accidental abstentions

The result looks odd until the scoring rule is visible.

A malformed observation scored correctness=false, review=true, and unsafe=false. It could not become an unsafe acceptance because it never produced a valid decision. The format failure sent the case to review.

Arm A had 105 valid observations and 61 malformed observations across the full schedule. Arm C had 166 valid observations and none malformed.

Compiled context did what it was meant to do. It removed structure failures. But that change also removed 61 accidental review decisions. More outputs reached the judgment stage. Some were correct. More were also unsupported accepts.

Malformed JSON is not a safety control. A system should not rely on broken output to stop an action. But the comparison exposes a real dependency: if a malformed response currently forces review, fixing the format changes the safety behavior of the whole pipeline.

The compiler improved the answer channel. It did not improve the permission policy.

What held up under scrutiny

There was exactly one completed attempt. Independent runner verification regenerated the summary byte for byte with SHA-256:

bea16fbb9fe5de39bb128be76f3874a3faecdc3c59c3cc7073743afa259130f4

The execution report SHA-256 was:

1fcf50d28d4dadb3997de77eff88db918b9e06e52d177aa9dae1c4b034913e7b

A custody recheck preserved the bound result. The repeat subset covered 48 arm pairs. Decision agreement, route agreement, malformed agreement, and evidence-ID Jaccard were all 1.0. That is realized repeatability for this subset, not general determinism.

Leave-one-base-out checks kept both movements in the same direction for every omitted base:

Threshold development selected no-safe-auto-action. Confidence was unavailable, so the study did not calibrate confidence.

Arm C used 2.4622 times the input tokens. Mean latency rose by 3.06%. Both numbers are descriptive. They are conditioned on the quiescent host and are not clean causal performance results. The quality phase captured no Metal trace.

The failed starts are part of the result

The run history matters because each stop protected the one-attempt rule.

V1 became invalid during adapter startup before observation 1. It recorded zero model loads and zero observations.

The first v2 preparation stopped at readiness because Colima met or exceeded the frozen 1 GiB unrelated-process limit. The second stopped during adapter-check because a Go host snapshot encoded empty arrays as null, while the strict Python binding required [].

Both v2 stops happened before authorization, model load, and inference. Neither consumed the attempt.

The producer fix changed the binding, not the protocol or scientific identities. It was outcome-independent because no model outcome existed. Reviewers merged the correction before a new authorization. Then one attempt ran to completion.

That chronology is not setup drama. It is evidence that the method failed closed.

What this result does not show

This result does not show that compiled context beats a larger model. No larger model ran.

It does not show that more context tokens caused the correctness or safety changes. The arms changed context construction, and the token and latency measurements remain descriptive.

It does not establish a scaling claim, calibrated confidence, general model determinism, production safety, or an unconditional benchmark. It does not transfer beyond the pinned local Qwen3-4B, the frozen A and C contexts, and this schedule.

A model that answers every time still needs permission to say no.

Evidence: Corrected implementation PR #109 · Exact merged implementation · Frozen v2 preregistration · V1 status and v2 amendment · Public evidence draft PR #110 · Public aggregate · Result report · Public evidence checksums