The Discovery Chain — Efficient Thinking VIII

Experience can improve a frozen intelligence's search. More self-generated experience does not automatically compound. The bottleneck is at least partly where and how experience is acquired. And the selection of that experience afterwards, prompt retrieval of it, and steering from its label are not where the value is.
Every number on this page is a measured result from a paper or a registered plan entry, with its section named. Nothing here is a sketch of what a result might look like.

VIII
Use
→
VIII-b
Accumulate?
→
VIII-c
Acquire
→
VIII-d · e
Select? Retrieve?
→
VIII-f
Three families
→
VIII-g · h
Steer? Which label?

The claim, in one sentence. When intelligence is held fixed, the value of experience depends strongly on how that experience was acquired, not merely on how much experience is stored or how it is retrieved. Each clause is a measured result: how it was acquired (VIII-c: the same head on states from a forced first look, 41.2%, against the agent's own experience, 26.8%, and base, 22.0%); how much is stored (VIII-b: a second generation of the agent's own experience buys a gain that does not separate from the loop's noise); how it is retrieved (VIII-d: no selection rule beats a random draw from the same pool; VIII-e: prompt retrieval of it is a null, while the read-only head gains 13 to 15 points). Scope, stated so it cannot be taken away: the acquisition result is measured on one model family and one task family; what replicates across three families is the head's use of experience, not yet the acquisition effect.

The method, in one sentence. This series treats the model's parameters as its thinking and never changes them; everything else that can move search — what the model is given to read, where that experience is collected, how it is applied at a decision — is varied one at a time and measured. Holding the thinking fixed is what lets each gain be attributed to experience rather than to a changed model, and it is what makes acquisition and use separable, measurable variables. The frozen-model-plus-readout half has neighbours (linear probes, contextual bandits over frozen states, hidden-state reward models; Paper VIII §6a). The separation of acquisition from use under a frozen intelligence, and the finding that acquisition is where the value moves, are this series' own.

Where in the chain this works. The head does not choose a final answer from information already computed; it decides where the next inspection goes, which is where the next unit of computation goes, and that changes what the system subsequently sees. In that sense it is a learned search prior, closer in kind to the selection rule of a tree search than to a learned action classifier, and its measured effect is on allocation: equal quality at 2.32× less compute (Paper VIII §7.6). The closest published readouts learn which action to take in the state the agent has encountered; this series learns how verified past experience should redirect future search, and VIII-c moves one step further up: the policy used to acquire that experience determines how useful the resulting search prior is. The chain the series measures is therefore acquisition policy → experience → search policy → computation → outcome, with the intelligence fixed at every link. Paper I showed that better-allocated search can substitute for model size; VIII asks whether accumulated, verified experience can make that allocation better without touching the model, and VIII-c asks what experience to collect so that it does.

What the method risks, and how that is measured. Changing a model's parameters to improve it carries a cost that is hard to quantify: every regression suite the model passed was passed by the old parameters, and what the new ones lose is not known until it is found. This series never changes a parameter, so that cost is zero by construction, and every suite the vanilla model passes it still passes (Paper VIII, registered prediction P5, holds by construction). The method has a risk of its own, and it is stated here because a reader should ask: the head chooses among candidates the model itself proposed, so it can steer search away from a place where the vanilla model would have succeeded. That loss is measurable, problem by problem, on the same problems: how many the base solves that the head does not. Paper VIII §7.6 reports it at the equal-quality point — 24.0% both, paired on 300 problems, 33 discordant each way — so at that budget the head's losses equal its wins and the gain is all in compute. Measured across the closed arms (2026-10-05, derived from the episode files, every published mean reproduced): the regression cell runs from 1.2% of problems (family 1 at 4-bit, 11 of 900) to 12.2% (one label on family 1 at bf16, 110 of 900), and it does not track the headline gain — the largest gains carry the smallest regressions. Against the stricter denominator, the problems the vanilla model itself solved, those same cells are 11.7% and 59.8%; both denominators are published, and the second is the one that prices the behaviour. Every arm publishes that four-cell table (both solve · head only · base only · neither) beside its headline, so the regression is a number on the page, never an inference. A head is fitted on one task family's verified history and, in every measured result, applied only within that family — the second search structure in Paper VIII got its own head, not a carried one — so the regression is bounded to the family where it is measured, and outside it the vanilla model runs unchanged. What a head does when applied outside its family has not been measured; it is registered as the next control, so that the per-family rule rests on a number rather than on caution.

Where this sits. The efficient-reasoning literature (length-controlled training, latent compression, confidence-steered inference) and this series ask one question: how much capability a fixed intelligence yields per unit of compute, and what moves that number. That work moves it from the compute side, by making a model think in fewer tokens. This series moves it from the experience side, by holding the weights frozen and giving the model its own verified past to read at each decision. Two methods on one axis; the second is the one this series measures.

The first three steps

VIII — Experience Priors

Can a frozen model use its own verified past to search better, without changing a weight?
Yes. A read-only linear head over the frozen model's hidden state, fitted on verified history, moves the quality–compute frontier.
081624324022.3%basegiven the head's compute31.7%frozen headon verified history

Matched compute: the base given the head's compute reaches 22.3% where the head reaches 31.7% (+9.3 [+3.7, +15.0], p 0.002). Equal quality (24.0% both) at 2.32× less compute. Four n = 300 measurements: +11.3, +13.0, +14.0, +13.3 problems in 100, all p < 10⁻⁴. Six other carriers of experience fail the same constraint. Replicates on a second model family (+14.3 [+7.7, +21.0]) and on a second search structure, SQL repair (+12.3 [+6.7, +18.0]).

Paper VIII §7.3–§7.7 · read

VIII-b — Does Experience Accumulate?

If the head learns from the model's own past, does a second round of that past help again?
Mostly not. A second generation buys a small, consistent gain that does not separate from the loop's own noise.
081624324022.0%baseno experience26.8%generation 1own experience

Generation 1 over generation 0: six of six paired games positive; match 1 mean +3.0 [+1.8, +4.2], match 2 mean +5.1 [−3.2, +13.4]; not established against the within-arm spread. The diagnosis: 49% of the second generation's training rows were byte-identical to the first's. The model keeps returning to the same places, so its own experience stops being new. (The 26.8% is the matched second-generation head as scored in VIII-c.)

Paper VIII-b (draft v0.3) · read

VIII-c — Where the Experience Is Acquired

Does where and how the experience is collected change its value, with the model and the head unchanged?
Yes, and by a lot. The same head on states from another agent's places, or from a forced first look, breaks the plateau.
0102030405022.0%base26.8%ownexperience35.4%other agent'splaces41.2%forcedfirst look

On 300 unseen problems: base 22.0%, own experience 26.8%, states from where a weaker second agent went 35.4%, states collected under a forced first look 41.2%. Reproduced from scratch on a second machine: +18.55 against the published +19.22, inside the bar. The intelligence did not change; the acquisition policy did.

Paper VIII-c §3–§5 · read

What came after, and what it ruled out

VIII-d — Selecting Experience

Given a pool of forced-look states, does a clever rule for choosing which 132 to fit on beat a random draw?
No. Coverage, readability and a mix-matched control are indistinguishable within ~5 points, and none beats a random draw from the same pool.
0102030405041.0%random drawsame pool39.3%readabilityrule39.0%coveragerule38.7%mix-matchedcontrol33.4%publishedanchor

Mean green over three games. The +3 to +8 that the selected heads showed over the published anchor was the pool, not the selection: a random 132 from the same pool scores 41.0 against 39.3, 39.0 and 38.7 for the three rules. Registered before each draw; nine of nine pairings negative; no interval excludes zero.

Series plan, VIII-d arm 1 and R5 · plan

VIII-e — Retrieval Against the Head

Is the head just a cheaper way of putting past successes into the prompt?
No. Prompt retrieval of past experience adds nothing and costs 5.9% more input; the head beats it by 13–15 points in every game.
081624324020.4%base19.3%promptretrieval33.1%frozenhead

Mean green over three games (base 20.4, retrieval 19.3, head 33.1). Retrieval minus base: −1.0, −2.0, −0.3. Head minus retrieval: +13.7, +13.3, +15.3, both criteria, three of three. Between the verifier label and the agent's-own-pick label, both help and their difference (+1.7, +4.0, +4.3) is below the ~6.7-point resolution; a 23-point offline advantage for one label converted to no in-loop advantage.

Series plan, VIII-e arm 1 · plan

VIII-f — A Third Family

Does the finding survive a third, unrelated model family, at 4-bit?
Yes. The head beats base by +21.3, +22.0, +23.0 and beats retrieval by +20.7, +22.0, +22.3, three of three; retrieval against base is a null.
0102030405015.6%base16.0%promptretrieval37.7%frozenhead

Yi-1.5-9B-Chat, 4-bit, head at layer 31 by the original's depth fraction, registered before any compute, scored after an independent second read with every artefact hash recomputed. Mean green over three games: base 15.6, retrieval 16.0, head 37.7. The verdict transfers across families; the size does not, and rates are never compared across families.

Series plan, VIII-f · plan

VIII-g — Steering From the Verifier's Label

If VIII's steering failure was construction rather than application, as the adaptive-steering authors suggested, does the success signal separate in hidden space?
No. The precondition is not met; the candidate ended at its first gate, in 0.85 hours.
012340.74verifier labellayer 180.77verifier labellayer 272.69agent's picklayer 182.92agent's picklayer 27

Within-task d′ of the verifier label at layers 18 and 27: 0.74 and 0.77 against a bar of 2, and less separable than the agent's own pick (2.69, 2.92 on the same 49 tasks). Not in conflict with VIII-f: the within-task rule removes the between-task structure a cross-task head can use.

Series plan, VIII-g · plan

VIII-h — What Should Experience Encode?

The verifier's label, or the agent's own behaviour? The VIII-e label pair re-run on nine disjoint task sets with a pooled estimator, criterion made symmetric before any count was read.
The verifier label wins: +16.6 points over the agent's-own-pick head, interval [+14.6, +18.5], nine of nine games, 3.6 times the bar. Useful experience carries information the agent did not already hold in its own behaviour.

Eighteen games at exactly 300 episodes, 67 artefacts archived and verified, criteria fixed before any count was read.

What each outcome would mean, written before the lock is opened: if the verifier-label head wins, useful experience carries information the agent did not already hold in its own behaviour; if the agent's-pick head wins, experience works by consolidating what the policy already knows (that head reproduces the agent's own choice about seven times in ten); if they are indistinguishable at nine draws, the label matters less than the existence of the channel itself.

Series plan, VIII-h · plan

Three separable variables

Intelligence the frozen weights; never changed in any arm.
Experience acquisition where the agent goes to collect its past; the variable that broke the plateau (VIII-c), and the one that selection rules cannot improve on afterwards (VIII-d).
Experience use how the past guides search; a read-only head over hidden state, on three families (VIII, VIII-f); not prompt retrieval (VIII-e); not steering from the verifier's label (VIII-g).

Author: Louay Alsakka · Efficient Thinking · the challenge page · github.com/LouayAlsakka/efficient-thinking

Site build after e567b61 · 2026-10-07 13:27Z