Qualifying Pre-Evaluation Confidence in LLM-Guided Combinatorial Codeword Search: An Evidence-Bound Measurement Audit

Abstract

Before an LLM’s verbalized probability can allocate search in adaptive combinatorial codeword search, the predicted event, reliability, ranking discrimination, incremental predictive value, stability, and design-point outcome support must be qualified. We present an evidence-bound measurement audit of one insertion–deletion codeword-search pilot, not a software-program-synthesis benchmark. The executed target was the joint event Y=V∧E: structural validity and fixed-reference exceedance; the realized data could not separate legality knowledge from exceedance knowledge among valid candidates. Temperature scaling reduced observed calibration error on one shared 87-candidate table. Because the transform was strictly monotone, its unchanged AUC of 0.567132 was analytical rank invariance, not a discovered ranking effect; six calibration seeds on those rows were computational sensitivity conditions, not independent replications. Across two unmatched pilots, we observed stage non-exchangeability and a saturated design point; a fully accounted downstream run produced 143/143 positives, leaving the preregistered subgroup decision non-estimable under the realized outcome and cell support. The five gates are a retrospectively instantiated minimum checklist proposed for prospective fail-closed use, not a validated universal protocol. Measurement qualification must precede any controlled test of confidence-guided search, and engineering completion must remain separate from scientific qualification.

1. Introduction

LLM-guided adaptive search can repeatedly propose, verify, and select candidate objects. In our case those objects were sets of binary codewords for an insertion–deletion coding task, not general source programs. The loop adapted through candidate history and verifier feedback without gradient updates to the model. Executable feedback revealed performance after evaluation, but a search policy would have to allocate calls, prune candidates, or redirect exploration before that feedback existed. Verbalized confidence appeared to offer an inexpensive control signal at that point. The temptation is to calibrate the score, observe that predicted probabilities align more closely with empirical frequencies, and then use it as if it also identified which candidate should be tried next.

That inference is unsafe because calibration and ranking discrimination answer different questions. A score can match an event’s base rate while assigning nearly the same value to every candidate; conversely, a useful ordering can be badly scaled as a probability. Even a score that is both calibrated and discriminative may target the wrong event, drift as the search changes its own candidate distribution, or be evaluated at a design point where almost every candidate has the same outcome. Decision utility is a further question: a qualified measurement still needs a controlled intervention before one can claim that acting on it improves search.

We therefore treat pre-evaluation confidence as a measurement that must pass five distinct, non-compensatory gates. Target validity asks whether the predicted event represents the intended decision and whether legality is separated from performance. Reliability asks whether probability levels match observed frequencies on the target population. Discrimination and incremental predictive value ask whether the score orders candidates and adds signal beyond inexpensive baselines. Stage stability asks whether those properties survive the distribution shift induced by adaptive search. Design-point outcome support asks whether event prevalence and subgroup occupancy permit the planned statistic and decision rule to be evaluated. Failure of any required gate blocks search-control interpretation while preserving the result as design evidence.

We apply this audit to one bounded, evidence-traced combinatorial codeword-search pilot with a fixed deterministic reference. The agent reported a probability, before verifier execution, for the joint event that a candidate would be structurally valid and would exceed that reference. Versioned plans, retained invalid runs, run-scoped manifests, confirmed validation records, and a sealed Evidence Pack make the path from protocol to manuscript claim inspectable. They do not make the verifier equivalent to human intent, remove adaptivity introduced by troubleshooting, or turn a single model/provider and testbed into a population of LLM-guided search systems. The case study is exploratory by design.

The central empirical pattern separates reliability from observed ranking discrimination. Post-hoc temperature scaling reduced calibration error on the realized table; because the transform was monotone, it analytically preserved the raw ordering and candidate-ranking AUC. The study also exposed distinct observed failure modes: the executed joint target was dominated by validity status and did not support the conditional exceedance question; outcome rates varied over search stage without establishing the preregistered monotonic trend; and a later design point produced an all-positive population. That downstream run reconciled every planned invocation, yet its missing outcome class and sparse planned intersections left the preregistered subgroup decision non-estimable. Execution completeness and scientific qualification diverged.

This paper makes three contributions. First, it proposes a minimum qualification checklist covering target validity, reliability, discrimination and incremental predictive value, stage stability, and design-point outcome support. The checklist is retrospectively instantiated on Cycle 1 and is proposed—not yet validated—for prospective fail-closed use. Second, it reports an evidence-bound combinatorial codeword-search pilot in which observed calibration error decreased while weak observed discrimination remained fixed under a monotone transform; this separates empirical scale repair from analytical rank invariance rather than claiming that confidence is absolutely uninformative. Third, it documents distinct failure modes in which joint-target ambiguity, stage association, design-point saturation, and outcome collapse left planned questions exploratory or non-estimable despite a successfully executed pipeline. The versioned provenance chain supports those contributions by retaining amendments and failures, but it remains a supporting audit mechanism rather than proof of validity.

The implications are procedural rather than a verdict on confidence-guided search. Measurement qualification should precede any frozen intervention comparing a confidence-guided policy with a matched control. Cycle 1 did not run D1/H3, so its absence is a scope limitation and provides no evidence about search utility in either direction. In retrospect, the unresolved target and outcome support show what a successor should qualify before an intervention; they are not asserted as the historical reason D1/H3 was deferred. The study does not establish external-feature superiority, human-ground-truth labels, or a general property of LLM-guided search. Section 2 locates the work relative to confidence calibration, tool-use, adaptive search, verifier validity, preregistration, and negative-result reporting. Sections 3 and 4 define the proposed audit and case boundary; Section 5 reports the evidence; Sections 6 and 7 derive protocol implications and limitations; and Section 8 closes with the prospective redesign required before utility can be tested.

2. Related Work

2.1. Confidence calibration in code generation

Prior work already treats model confidence as a measurable probability in programming tasks. Toward Trustworthy Neural Program Synthesis estimates the correctness probability of sampled programs, while Calibration and Correctness of Language Models for Code evaluates and post-hoc calibrates confidence for source-code generation. This line now extends beyond one-shot synthesis: Fine-grained Approaches for Confidence Calibration of LLMs in Automated Code Revision studies program repair, vulnerability repair, and code refinement. Work on verbalized confidence likewise shows that elicitation and calibration depend on prompting and evaluation setup. These are methodological neighbours, not claims that our binary-codeword testbed is a software-program-synthesis benchmark. Our study therefore neither introduces confidence calibration nor assumes that a calibrated score is automatically informative.

2.2. Calibration, ranking, and action are different questions

Calibration concerns probability scale, whereas discrimination concerns ordering and incremental predictive value concerns performance beyond specified baselines. This distinction is explicit in Multicalibration for LLM-based Code Generation, which also illustrates how low calibration error can coexist with a low-information predictor, and in evaluations that report calibration alongside AUC. Related agent work has already connected confidence to behavior: Uncertainty Calibration for Tool-Using Language Agents uses calibrated internal probabilities for prompt and trace selection; MICE for CATs uses expected tool-calling utility to decide whether to act; and verbalized confidence can trigger self-verification. Confidence-Orchestrated Self-Evolution further uses intrinsic confidence to modulate policy updates and replay. In operations research, COOPA generates multiple candidate formulations, assigns 0–100 confidence scores across four modeling dimensions, and uses a max-min criterion to select one before solver execution. Its ablations test relative selection value through first-candidate comparisons, confidence-gain associations, and beneficial versus harmful overrides, but do not estimate absolute calibration of a specified-event probability. Thus, neither confidence-to-action nor empirical testing of confidence-guided selection is our novelty. We instead ask what joint evidence a specified-event pre-evaluation probability must satisfy before a controlled search-use experiment is warranted; our pilot did not execute that intervention.

2.3. Tool regimes, evolutionary search, and verifier validity

The application class itself also has direct prior art. LLM-Guided Search for Deletion-Correcting Codes adapts FunSearch to evolve executable priority functions that construct binary deletion-correcting codes, evaluates the resulting code sizes, and studies deduplication, generalization, and search-design choices. Our search representation differs—the measured candidates here are codeword sets rather than priority functions—but LLM-guided deletion-code construction is therefore not our novelty. More broadly, LMEA uses an LLM for parent selection, crossover, and mutation in combinatorial optimization; Language Model Crossover evolves binary bit strings among other text-representable genomes; and a stage-wise reliability study of LLM evolutionary optimizers evaluates constraint fidelity, repair, cost, and model dependence. COOPA additionally establishes that verbalized confidence can select pre-execution optimization candidates and that its relative selection signal can be tested. We did not find a prior study that jointly qualifies a frozen specified-event probability through the same target-validity, absolute-reliability, discrimination, stage-stability, design-point-support, and provenance gates.

Confidence behavior is also regime-dependent. The Confidence Dichotomy finds that calibration changes across multi-turn tool settings and distinguishes deterministic verification feedback from evidence-retrieval tools. In self-evolving systems, adaptation and verification are established concerns: AdaEvolve changes strategy under stagnation, Socratic-SWE converts execution traces into targeted training tasks, and The Verification Horizon studies verifier proxies, reward hacking, and signal saturation. ACE further uses adversarial unit tests when stronger solvers make ordinary verifier tests less failure-revealing. These works preclude a claim that stage dependence, optimizer reliability, or verifier saturation is new. They do not, however, make a deterministic verifier equivalent to human intent. Research on task indeterminacy and self-referential evaluation validity reinforces the need to audit what an evaluation target represents.

2.4. Protocol-first and negative-result reporting

Neither preregistration nor transparent negative-result reporting is new to machine learning. The NeurIPS Workshop on Pre-registration in Machine Learning implemented protocol-first review followed by execution and confirmatory review, while Embracing Negative Results in Machine Learning argues directly for publishing informative failures. Our contribution is therefore not the existence of a preregistration record or an inconclusive outcome. It is the use of prospective qualification states and retained provenance to distinguish a measured deficiency from an untested or non-estimable question.

2.5. Position of this study

To our knowledge as of 27 August 2026, prior work had already applied LLM-guided evolutionary search to deletion-correcting-code construction and used verbalized confidence to select pre-execution optimization formulations, but we did not find a study that jointly qualified a frozen specified-event probability through the same target-validity, absolute-reliability, ranking-discrimination and incremental-value, stage-stability, design-point-support, and provenance audit. The contribution is a proposed minimum qualification checklist plus an evidence-bound case study and transparent scientific cliff—not LLM-guided code construction, a new calibrator, a first use of confidence for adaptation, a first test of confidence-guided candidate selection, a software-synthesis benchmark, or a general law about LLM-guided search. Because this is a structured rapid review rather than a systematic review, the boundary is cutoff-specific even after independent review and must be refreshed again when a venue is selected or more than 30 days elapse before submission.

3. Measurement-Audit Protocol

Pre-evaluation confidence should enter search control only after qualifying as a measurement of a deployment-relevant event. We propose five distinct, non-compensatory gates: target validity, reliability, discrimination and incremental predictive value, stage stability, and design-point outcome support. Their strengths cannot be averaged because each guards a different source of uninterpretable search behavior.

The checklist is designed for prospective, fail-closed use, but Cycle 1 did not validate it prospectively as a unified protocol. We retrospectively instantiate it on retained Cycle 1 evidence and propose that successor studies freeze it before collection. Each load-bearing gate receives one of four states: pass, fail, not estimable, or not tested. Only a pass supports progression to a controlled search-use experiment. A fail identifies a measured deficiency; the other unresolved states mean the planned decision could not be evaluated or a qualification-grade test was not completed, not that the capability is absent. Exploratory evidence may therefore be reported under not tested without being promoted into a prospective pass. Figure 1 summarizes this proposed logic, and Table 1 states the required evidence categories.

Table 1. Five-gate qualification protocol

The gates are logically distinct, but audit order should reduce wasted evidence. Target validity and design-point support preflight precede main collection; reliability, discrimination, and stage stability then operate on sealed predictions and outcomes. A deployment policy can make a gate especially load-bearing: absolute thresholds depend on reliability, ranking allocation depends on discrimination, and persistent policies depend on stage stability. Even when a gate is not load-bearing for a narrow policy, its status should be reported rather than omitted. Passing a downstream diagnostic cannot repair upstream target invalidity or inadequate outcome support.

The checklist specifies an interface for preregistration rather than universal cutoffs. For each gate, a successor study must freeze the target estimand, independent unit, input population, diagnostic family, study-specific acceptable region, uncertainty and multiplicity rule, missingness/attrition treatment, state transition, and redesign action. Those values must be justified for the proposed search policy before outcomes are observed; this paper does not prescribe one ECE, AUC, alpha, sample-size, or occupancy threshold for every task.

3.1. Target validity precedes score evaluation

The intended event must specify what is predicted, prediction time, decision horizon, comparator, tie handling, and validity constraints. An executable verifier labels that operational event; it does not automatically represent human intent, usefulness, or scientific validity. If success combines validity with threshold exceedance, the audit should also measure validity and exceedance conditional on validity. Otherwise, validity cues alone can mimic performance knowledge. Labels must follow score sealing, and prompt, feature, and artifact review must exclude target leakage.

3.2. Reliability is necessary but insufficient

Reliability asks whether cases assigned probability p experience the event at frequency p on the target population. The audit combines reliability diagrams, a proper score, calibration-error sensitivity, and low-information controls such as constant or rolling-base-rate predictors. Those controls can be calibrated under stable prevalence without candidate-level ranking. Post-hoc calibration may correct scale, but a monotonic mapping cannot create a new ordering. Passing authorizes probability-scale interpretation only, not discrimination, incremental predictive value, or utility.

3.3. Ranking and incremental predictive value require separate evidence

Discrimination asks whether higher scores precede positive outcomes. Incremental predictive value asks whether the score improves out-of-sample prediction beyond frozen history, search-state, and candidate-feature baselines. The protocol reports a ranking statistic, a paired proper-score or nested predictive comparison, and uncertainty over independently generated candidate populations. Repeated calibration fits on one table are computational, not empirical, replicates. A score failing this gate should not govern prioritization, pruning, restart, or resource allocation.

3.4. Search stage is part of the data-generating process

Even with a frozen event, iterative search changes prompts, parents, candidate quality, and history. A global calibration map or threshold may lose validity across stages. The audit compares stage-conditional prevalence, reliability, ranking, and residuals; checks drift or exchangeability; and predefines evaluation of online or regime-specific calibration. Association alone neither proves monotonic degradation nor identifies cause. Until dependence is modeled, restrict the score’s validity interval rather than projecting it across the trajectory.

3.5. Design-point outcome support is a design gate

Before the main budget, a blinded design-point preflight should test for both outcome classes, support for the conditional event, and planned subgroup occupancy. Its rule, acceptable region, independent unit, and redesign action must be frozen before main outcomes are inspected. More calls cannot recover discrimination from a single-class population, and more bootstrap replicates cannot populate empty cells. Failure requires a successor protocol changing task difficulty, reference, sampling, cell structure, or estimand. This gate concerns realized outcome and cell support for a specified decision; it is not a blanket claim of structural parameter non-identifiability.

3.6. Qualification and reporting

Engineering completion and scientific qualification are separate. Runtime reconciliation, immutable artifacts, and version history establish what ran; they cannot convert a failed gate into a pass. Search control becomes eligible only when every policy-relevant gate has prospective evidence and an explicit pass. Post-failure amendments create a successor protocol and retain the prior result, allowing repair without presenting a revised estimand as the original test.

4. Evidence-Bound Case Study

4.1. Testbed, target event, and frozen reference

We studied one bounded insertion–deletion coding-theory task with a deterministic executable verifier. A candidate was a set of binary codewords at the selected n=11, k=2 design point. This was combinatorial codeword search, not general source-program synthesis. The verifier checked the pairwise longest-common-subsequence structural constraint and computed candidate size. Let V denote structural validity and E denote strict exceedance over the frozen algorithmic reference. The event elicited before verification was the joint event Y=V∧E; ties were negative. “Pre-evaluation confidence” therefore denotes a candidate-level verbalized probability for this executable joint event, not a general report of self-knowledge or correctness.

The reference was produced by a fixed-budget PGCS procedure and frozen at size C_base=12. It was chosen to create a potentially mixed outcome distribution, not because it approximated the insertion–deletion converse bound or human intent. The deterministic verifier was authoritative for Y only. A negative outcome could arise because V=0 or because V=1 and E=0. The conjunction was part of the executed target and is not redefined after observation. However, the scientific question needed for performance allocation among legal candidates is conditional exceedance P(E|V=1). Cycle 1’s realized support could not separate that conditional question from legality recognition, making the executed joint target construct-misaligned for that downstream decision.

4.2. Executed scope and candidate populations

Cycle 1 executed two evidence-bearing experiment families. D2 collected one table of N=87 candidates from six generation seeds, with confidence recorded before deterministic evaluation. Its later 8 arms × 2 label modes × 6 calibration seeds analysis reused those same candidate rows; the 96 analysis trials were computational conditions, not independent candidate populations. D2 addressed H1 and part of H2. The intended D1 experiment for H3—whether confidence-triggered redirection outperformed a stagnation counter—was deferred and never run, so Cycle 1 contains no intervention evidence for H3 and licenses no utility conclusion in either direction. The retrospective audit motivates qualification before a successor intervention but is not evidence about why the original defer occurred.

D3 was a separate exploratory pilot at a different task/reference design point. Its collection design committed 144 model invocations and sealed N=143 candidates for downstream annotation and descriptive subgroup analysis. D2 and D3 were neither randomized against each other nor matched as a controlled design-point experiment. Both used one tested generation model family through one provider runtime on the same insertion–deletion testbed. The D3 annotation panel used a different model/provider from the generator, but its logical seats shared one panel model and were separated only by fresh, vote-isolated contexts.

4.3. Estimation and annotation boundary

D2 compared raw verbalized confidence (L1), a running-climatology baseline (L0b), monotonic temperature scaling (L2), three richer calibration constructions (L3), and an external-feature-only sensitivity probe. Reliability was summarized with equal-width ECE, debiased ECE, and Brier score; ranking was described with AUC. Paired sign tests were computed across the six calibration seeds, which reused the same 87-candidate table and were therefore not independent empirical replications. The recorded multiplicity correction was applied to the frozen comparison family. Stage analyses compared observed outcome rates and a three-stage trend diagnostic. H2’s larger planned estimator grid was not fully instantiated, so its evidence remains partial.

For D3, a deterministic rule assigned one of nine edit categories to every sealed candidate. A blind sample of 69 items received panel review. The released labels contained 62 locked panel consensuses, seven low-consensus pluralities, and 74 deterministic-rule labels that were not panel-reviewed. No human ground truth or human adjudication was collected. Accordingly, unweighted Cohen’s κ between rule labels and panel plurality measured rule–panel agreement under this protocol; it did not measure human inter-rater reliability or label accuracy. Subgroup gaps, occupancy, and whole-run bootstrap outputs were descriptive because the outcome and design cells did not support the planned H4 decision.

4.4. Provenance, amendments, and exploratory boundary

Each executed step wrote run-scoped artifacts, manifests, metrics, and protocol bindings. Plan revisions created successor versions while retaining failed, interrupted, superseded, and protocol-invalid history. Confirmed validation records linked the four hypotheses to their permitted verdicts, and a sealed Evidence Pack indexed the two final plans, ten authoritative runs, one analysis, and four validations. This hierarchy supports tracing a manuscript statement back to frozen bytes; it does not guarantee adequate outcome support, an evaluable decision rule, or a correct scientific interpretation.

One load-bearing D2 amendment occurred after v7 artifacts revealed that the nominal L2 skill ratio divided by an approximately zero L0b debiased ECE. The direct L2-versus-L1 endpoint was recorded before the v8 rerun, but after the failure became observable. We therefore report the replacement and its result as exploratory. D3 likewise retained an earlier protocol-invalid collection attempt and proceeded under a revised exploratory protocol with stronger runtime accounting. Neither repair retroactively restores preregistration. Together with the unexecuted D1/H3 branch, these boundaries define Cycle 1 as an evidence-bound case study rather than a confirmatory test of LLM-guided search systems in general.

5. Results

All results in this section describe one exploratory Cycle 1 case study. None is classified as confirmatory. Table 2 separates the observed statistic from the inference that the frozen evidence permits.

Table 2. Compact result summary

Table 3. Cycle 1 qualification-gate dispositions

Because Cycle 1 spanned D2 and D3, this table reports the most restrictive disposition relevant to progression into search control, not a claim that every diagnostic in every run had the same state.

5.1. Calibration error decreased; rank invariance was analytical

The raw verbalized score was overconfident at the studied D2 design point. Across the pooled 87-candidate table, the candidate-weighted mean prediction was 0.789885 while the observed joint-event rate was 49/87 = 0.563218. The L1 raw arm had equal-width ECE 0.262759, debiased ECE 0.165676, and Brier score 0.299202. Temperature scaling reduced the L2 debiased ECE to 0.036403 and the equal-width ECE to approximately 0.1138. Under the revised direct L2-versus-L1 endpoint, the recorded BH-adjusted sign-test value was p=0.0234375, and the finite-sample Venn–Abers diagnostic passed. Because all six calibration seeds reused one candidate table, that p-value is a computational sensitivity summary, not a population-level probability bound or six independent replications.

This calibration result is exploratory rather than confirmatory. The nominal preregistered endpoint divided by the L0b debiased ECE, which was approximately zero and therefore made the ratio skill undefined. The direct L2-versus-L1 comparison was frozen before the v8 rerun, but only after v7 artifacts had exposed the denominator failure. The revision resolves an invalid statistic; it does not restore prospective confirmatory status.

Temperature scaling was monotonic, so unchanged rank order and AUC were mathematical consequences of the chosen transform rather than discovered empirical effects. The mean AUC was 0.567132 for both L1 and L2. The raw and BH-adjusted sign-test values for L2 AUC above 0.5 were 0.015625 and 0.0234375, respectively, but the six calibration seeds reused the same 87 candidate rows. They are not six independent candidate populations, and the repeated sign does not license a confirmatory discrimination claim. The L0b running-climatology comparator had observed AUC 0.567669; the small point difference from L1/L2 is likewise not treated as a confirmatory comparison. L2 Brier skill relative to L0b was only +0.01469, with raw p=0.109375.

This no-gain claim is confined to the frozen monotonic L1-to-L2 scaling comparison. The per-seed best-L3 construction had a higher observed mean AUC (0.617660), but that post-selected summary reused the same rows and its three component arms had negative Brier skill; it is an exploratory construction, not evidence that the calibration procedure improved ranking (§5.5; Appendix B.3).

The supported empirical result is therefore narrower than “confidence was uninformative”: calibration error decreased on the realized table, while raw observed discrimination was weak and monotone scaling could not repair it. Calibration alone did not establish ranking gain, incremental predictive value, generalisation, or decision utility.

5.2. Candidate validity and reference exceedance were nearly confounded

The executed target was the joint event Y=V∧E, but the scientific distinction needed for performance allocation was whether confidence identified E among candidates with V=1. The realized joint outcome was strongly aligned with validity. Of 87 D2 candidates, 51 were valid and 49 satisfied the joint event. Of the 38 negatives, 36 were LCS-invalid, leaving only two valid negatives. Equivalently, 49/51 = 96.1% of valid candidates were positive and 36/38 = 94.7% of negatives were invalid.

This distribution makes two substantively different predictions difficult to distinguish: P(V=1) and P(E=1|V=1). An agent—or an external predictor—could obtain much of the observed discrimination for Y by recognizing surface indicators of validity without estimating exceedance among valid candidates. The result does not show that validity caused the confidence score, nor that every unit of discrimination came from validity. It shows that this dataset lacked enough valid negatives to evaluate the conditional performance question required by the intended downstream decision.

The confound changes the interpretation of every D2 discrimination result. A higher AUC for Y cannot, by itself, be interpreted as knowledge about performance relative to the reference among valid candidates. A future design must separately model P(V=1) and P(E=1|V=1) and must guarantee an adequate valid-negative sample before testing incremental predictive value.

5.3. Search stage and design-point support governed evaluability

Using the fixed reference did not make the candidate-generating process exchangeable over search stage. In D2, the early and late positive rates were 0.411765 and 0.611111, and the recorded exchangeability test gave p=0.0067. However, the estimated three-stage slope was 0.099673 with p=0.325. Thus the data support a stage association or non-exchangeability warning, not a monotonic early-to-late degradation claim.

The design-point comparison exposed a more severe measurement risk. D2 produced a mixed pooled base rate of 49/87 = 0.563218, although two individual generation seeds lay outside the target interval. D3, using the same fixed-reference concept at a different task/reference setting, produced 143/143 positives. With no negatives, realized-sample AUC is undefined and subgroup calibration cells cannot support the planned contrast. This D2/D3 contrast was not randomized or controlled and cannot establish that design-point choice caused the difference. It nevertheless demonstrates that a fixed reference does not structurally guarantee an informative base rate.

These observations motivate a fail-closed design-point preflight: before spending the formal study budget, a blinded pilot should verify a target base-rate interval, adequate valid-negative support, and minimum subgroup occupancy. Stage should be treated as part of the data-generating process, while difficulty and reference should be treated as design variables that can determine whether the planned statistic and decision rule are evaluable.

5.4. Complete D3 accounting coexisted with a non-estimable decision

The D3 collection run completed its engineering accounting. All 144 planned provider invocations were committed; 143 produced sealed candidates and one ended in schema-violation attrition. No systemic halt or process failure accounted for the outcome distribution. The conservation identity was therefore 144 committed = 143 sealed + 1 attrition.

The preregistered H4 decision was nevertheless non-estimable under the realized support. All 143 sealed candidates satisfied the fixed-reference event. Of 45 planned intersections, 38 were empty and three were sparse, leaving four usable cells (7 occupied − 3 sparse). Realized-sample rank discrimination was undefined because one outcome class was absent; conditional contrasts had empty or sparse support. The analysis produced a descriptive aggregate absolute gap of 0.074615 and completed its planned bootstrap computation, but the protocol correctly set confirmatory_inference_allowed=false and h4_verdict_allowed=false. Ten thousand computational replicates cannot recover a missing outcome class or empty design cells. This is a statement about the specified H4 decision under realized support, not proof that every population parameter is structurally unidentified.

The annotation chain also has a bounded interpretation. Sixty-nine items entered panel review; 62 reached locked consensus and seven used low-consensus plurality, while 74 other items retained deterministic rule labels without panel review. Rule-versus-panel plurality agreement had unweighted Cohen κ 0.165112 and exact agreement 21/69; raw disagreement 48/69 is derived from that exact count. These statistics describe isolated same-model panel consistency, not human ground truth, human inter-rater reliability, or label accuracy.

D3 is therefore both an engineering success and a pilot whose preregistered H4 decision was non-estimable under realized support. Reporting those states separately prevents a green runtime ledger from being mistaken for evidence that the planned estimand was evaluated. The confirmed H4 verdict remains inconclusive, not “no subgroup miscalibration.”

5.5. External features provided a stronger observed sensitivity signal

The external-feature-only probe had observed mean AUC 0.672932, compared with 0.567132 for L1/L2 and 0.617660 for the per-seed best-L3 construction. Its mean Brier score was 0.222431, versus 0.299202 for L1. Raw paired sign tests against L1 and best L3 were both p=0.015625.

These values are an exclusion signal, not a superiority result. The external comparison was removed from the final formal BH family, lacked a frozen paired DeLong interval or cluster bootstrap, and reused the same candidate population across calibration seeds. The “best L3” value additionally selects the maximum L3 arm within each seed before averaging. We therefore report that external features showed higher observed discrimination and motivated an incremental-value test; we do not claim statistically confirmed superiority or prove that verbalized confidence contains no residual signal.

The appropriate follow-up is a preregistered paired analysis that first predicts the separated conditional event from external features and then tests whether verbalized confidence adds out-of-sample predictive value. That analysis must occur after validity and exceedance have been separated.

6. Implications for Confidence-Guided Search

The case study supports a change in deployment order, not a claim that confidence-guided search is ineffective. A pre-evaluation probability should first be treated as a measurement under audit. Only after its target, probability scale, ordering, stability, and design-point outcome support have qualified should a separate intervention test ask whether using it improves search. This ordering prevents calibration alone from being promoted into a ranking, pruning, or redirection policy.

6.1. Freeze the decision target and preflight its support

The event definition should precede collection and include prediction time, comparator, tie rule, validity constraints, and intended decision. When success combines legality with performance, collection should preserve both components so that P(V=1) and P(E=1|V=1) can be evaluated separately. A blinded pilot should then check prevalence, valid-negative count, and planned subgroup occupancy against study-specific rules frozen in advance. Failure at this stage is useful design evidence: it calls for a harder or easier reference, different sampling, or a different estimand before the main budget is spent. It does not justify a capability verdict.

6.2. Match diagnostics to the proposed control action

Different search actions require different evidence. Absolute probability thresholds require reliability and proper-score comparisons. Prioritizing or pruning candidates requires out-of-sample discrimination and evidence that the score adds predictive value beyond inexpensive frozen baselines. A persistent policy across an adaptive search requires stage-conditional stability or an explicitly stage-specific model. Monotonic recalibration may repair the first requirement while analytically preserving rank order, as in Cycle 1. The evaluation should therefore report each gate separately rather than collapse them into a single “confidence quality” score.

6.3. Keep engineering and scientific ledgers side by side

Runtime completeness remains necessary. Invocation conservation, sealed inputs and outputs, protocol hashes, and retained amendments make failures diagnosable and reduce selective reporting. They do not, however, create missing outcome classes or occupied design cells. We recommend a dual status record: one field for execution integrity and another for scientific qualification. A run may be complete/qualified, complete/non-estimable for a specified decision, incomplete/recoverable, or protocol-invalid. Downstream analysis should unlock only when both the relevant engineering checks and the scientific preflight pass; unresolved states should retain their artifacts and route to redesign rather than disappear from the history.

This separation should also govern resource escalation. A small capability probe can verify that the runtime and output contract are observable; a blinded design-point probe can verify that the target population supports the planned estimand and decision; only then should the full collection begin. Halt reasons, attrition, and protocol amendments remain part of the estimand rather than being silently retried away. This staged unlock makes a failed preflight cheaper and more informative than a formally complete but unusable study.

6.4. Reserve utility claims for a controlled intervention

After the five measurement gates pass, the next experiment should compare a frozen confidence-guided policy with a matched control under a prospectively chosen utility metric such as best score per call, time-to-threshold, or compute-adjusted regret. Allocation rules, stopping criteria, and failure handling must be fixed before outcomes are observed. Cycle 1 did not run that intervention; this is a scope limitation, not evidence that calibrated confidence improves or fails to improve adaptive search. The retrospective findings instead motivate a prospective successor that qualifies target validity and outcome support before testing policy utility, without recasting that recommendation as the historical cause of the original defer.

7. Limitations and Threats to Validity

Construct validity. The executed joint event Y=V∧E was operational, not a complete representation of human usefulness or the conditional performance decision. In D2, validity and exceedance were nearly confounded, so observed discrimination for Y could reflect legality cues rather than performance knowledge among valid candidates. D3 edit labels combined deterministic rules with an isolated same-model AI panel; its κ and exact agreement measure consistency between operational label sources, not human ground truth, human inter-rater reliability, or accuracy. The external-feature probe likewise diagnoses a possible exclusion problem but does not identify what semantic information each predictor used.

Internal and statistical validity. The H1 primary endpoint was replaced after v7 output exposed a near-zero denominator. Although the direct comparison was frozen before the v8 rerun, this sequence makes the calibration finding exploratory. The six calibration seeds reused the same 87 candidate rows, so the sign tests describe computational sensitivity on one candidate-table realization, not six independent population replications. The external-feature comparison lacked a frozen paired DeLong interval or cluster bootstrap and was outside the final multiplicity family. Best-L3 selection was also post-selection within seed. D2 stage summaries are observational; the D2/D3 design-point contrast changed more than one condition and cannot support a causal attribution. H2 instantiated only part of its intended diagnostic grid. D1/H3 was never run, so the study contains no intervention evidence about confidence-triggered redirection. D3’s bootstrap output is descriptive because outcome saturation and sparse occupancy prevented the preregistered simultaneous decision.

The study also accumulated information through sequential troubleshooting. Failed and protocol-invalid attempts were retained, and revised versions were made visible, but later design choices necessarily benefited from observing earlier failures. That history is valuable engineering evidence and an explicit source of adaptivity; it is not equivalent to a single prospectively frozen confirmatory protocol. Reported p-values and confidence procedures must therefore remain attached to their exploratory or descriptive status rather than being read in isolation.

External validity. The evidence comes from one model family/provider configuration, one insertion–deletion combinatorial codeword-search testbed, one fixed-reference family, and a small set of search trajectories. It is not a software-program-synthesis benchmark. The observed validity mix, stage association, calibration response, and D3 saturation may change with model, prompt, task, verifier, reference difficulty, or search policy. Cycle 1 is therefore a case study of a measurement workflow and its failure modes, not a population estimate for coding agents, software generators, or adaptive LLM systems generally.

Reproducibility and reporting. Versioned plans, retained invalid runs, manifests, and the sealed Evidence Pack improve auditability but do not by themselves establish correctness. P7 now provides a review-candidate public boundary, path sanitization, hash verification, an evidence map, and a summary-level recomputation command. It has not completed third-party independent recomputation, snapshot-based code release, legal/privacy/security review, long-term archive, or submission binding; the candidate package is not approved for public release, so reproducibility remains a target rather than a completed claim. The related-work landscape is also moving quickly and must be refreshed within the planned submission window. Finally, future Cycle 2 evidence must be reported as a prospective successor, including contradictory results, rather than used to relabel this exploratory pilot as confirmatory.

8. Conclusion

In this evidence-bound combinatorial codeword-search pilot, calibration error decreased on one realized table, while weak observed discrimination remained fixed because the chosen transform was monotone; probability-scale repair alone therefore could not qualify the score for search control, but this case does not show that confidence is absolutely uninformative. The audit further separated construct misalignment between joint and conditional targets, stage association without an established monotonic trend, and a saturated design point where complete runtime accounting coexisted with a non-estimable subgroup decision under realized support. These were scientifically useful failure states, not falsifications of unexecuted or non-estimable hypotheses. A prospective successor should freeze V, E, and the intended decision, preflight base rate and subgroup occupancy, test incremental predictive value on independently generated data, and only after all five measurement gates pass compare a confidence-guided policy with a matched control, while continuing to report engineering integrity and scientific qualification as distinct outcomes and retaining contradictory results, failed preflights, and protocol amendments as evidence rather than silently overwriting them.