Agreement, Anchors, Addresses: A Week of Geometric Training

AbstractPhil · August 2026 · continues Geometric Memory FT4 - Distill Against a Consensus (and Choose Your Frame)

---

This week a 58-million-parameter text encoder, trained with only the agreement of five "teacher" models - existing encoders whose sole role is to say what answer they would give - beat every one of them on an 8-task sentence-similarity mean: .6077 against the best teacher's .5883, at 13% of the five teachers' 461M combined parameters, with every model read under one identical harness. The same week, a softmax-free attention mechanism went from passing its entire static mathematical battery, through a measured training collapse, to a converging prototype, with every step of that arc priced in numbers.

This is the record of that week, run by a team of two (a researcher and an AI collaborator) on rented GPUs and a pair of local cards, as part of a research program whose prior installment is linked above. The method throughout is a simple discipline applied relentlessly: every mechanism earns its place by measurement, and when two designs compete, an experiment arbitrates. Variants and invariants were tested in pairs; formulas were kept when they functioned and amended when the measurements disagreed. The pattern that emerges across six distinct lines of work is that geometric structure - agreement between models, anchor directions in activation space, signed addresses over tiny codebooks - keeps earning its keep wherever it is measured carefully: sometimes as a capability gain, sometimes as a free reorganization, and twice this week as a priced null.

Everything below is public. The models, adapters, benchmark JSONs, and training code live on Hugging Face under AbstractPhil; each model family additionally carries a technical companion document with the full numerics for readers who want depth beyond this article - every companion is linked directly in the artifacts table at the end.

---

0. Trying something a little different

Structurally, this article is meant to be a more narrative than the prior installments. The program's own notes are interleaved with the results, and the companion documents carry the full tables and per-run numbers. The goal is to give a sense of how the week unfolded, what was measured, and what was learned - not just what the final numbers were.

With the narration, the links to the more technical documents are meant to let readers skip the narrative and go straight to the numbers, or to read the narrative and then check the numbers. The companion documents are not a substitute for this article, but they are a supplement: they carry the full tables, per-run numbers, and the details of the training and evaluation environments.

As it stands, the documentation is becoming more and more difficult to keep track of. The program is now a family of models, adapters, and training recipes, and the companion documents are a way to keep the details organized. The goal is to make it easier for readers to understand what was done, how it was done, and what the results were.

With that, I've made this a much smaller article than the prior installments, and I've tried to keep the narrative focused on the key points. The goal is to give a sense of what was measured, what was learned, and the core implications and developments of the week, without getting lost in the weeds of every detail.

1. Two encoders trained on agreement alone

The week's foundation is a pair of sibling text encoders - captionbert-8192-v2 and captionbert-8192-v2-B - trained from scratch by consensus distillation. Five BERT-family teachers each embed the same ~27–32 million image captions. Each teacher receives one fixed alignment (a whitened-Procrustes map: a normalization plus rotation that lines its embedding space up with the others) into a shared frame. The training target for every caption is the normalized centroid of the aligned five. Teacher agreement is the entire supervision signal.

The headline: the student reaches an 8-task mean of .6077 - mean Spearman correlation across eight standard sentence-similarity benchmarks, a scale on which the best of its five teachers scores .5883 and a strong pair-trained model scores ~.79 - having seen zero similarity labels. For orientation, the strongest label-free contrastive methods sit in the mid-.70s on suites like this; the claim here is about what teacher agreement alone can supervise, not a leaderboard position. It settles a question the program had carried for weeks: an earlier 26M-parameter attempt at 500k rows had only matched its teacher median, and the conclusion then was that the effect had limits. At 58M parameters and 27M rows the effect switches on decisively. Both model size and data moved together between those two runs, so the boundary is bracketed rather than pinned to either axis - but above it, agreement outperforms its sources.

The sibling run asks the obvious follow-up: how much of this is data volume? Model B trained on the completed corpus, 19% more rows, and landed at .6031 - a .0046 difference, comparable to the program's measured seed-to-seed band. More captions bought nothing. The evidence points at teacher agreement as the ceiling: the consensus target occupies an effective rank of about 29 of 768 available directions (effective rank: roughly, how many independent directions the vectors actually use), and the reference teacher's own embeddings hold ~40 effective directions on the same corpus. A consensus built in that frame inherits a hard budget, consistent with a ceiling set by the frame itself - a design lever the program can now set deliberately, with teacher-set diversity as the untested axis.

One more property of the recipe matters for anyone deploying it: the choice of alignment frame decides what ships. Aligning the teachers to a reference member (BERT's own frame) causes the student's output space to drift into coincidence with that frame during training - the gap closes on its own - so the text models ship as plain encoders that drop into any pipeline. The program also runs this recipe on image models (the prior installment linked above); that consensus was aligned to a running mean instead, and it ships with a small rotation matrix that every consumer must remember to apply. Same mathematics, opposite shipping requirement, one instance of each measured.

Why these benchmark points matter

• .6077 vs .5883 (student vs best teacher): label-free distillation of agreement can exceed every individual source - the compression argument for consensus methods.
• .6031 on 19% more data: more captions bought nothing, so the productive levers are teacher-set diversity and frame choice.
• Rank ~29/768 target, ~40/768 teacher: capability ceilings here are geometric budgets, visible before training - measurable and plannable.
• On one of the eight tasks (STS12), every teacher scores .31–.43 and both students hold ~.50: the clearest single case of the consensus surpassing its sources where their scores scatter most.

2. Anchors that carry behavior: the adapter library

Aleph anchors were discovered and excavated through a large series of small experiments, many of which are heavily documented. There are multiple variants and invariants to the anchors, and there are multiple rules applied to each variant. The identification of the infinite state space of the anchors took a great deal of experiments to approach an analysis, and the math formulas involved were built specifically to excavate, measure, arbitrate for, align, normalize, standardize, and every other format of measurement that the program could think of. Many of the experiments failed, and the survivor list was very small. This structure we're utilizing here, is the result of the seeded survior. One of the few that actually made it beyond the initial stages and managed survived every major battery since, hence this is the lineage of the emergent Aleph mechanism.

A word on the mechanism at the heart of everything that follows. An anchor is a learned direction in a model's activation space. Tokens are scored against a small set of anchors by a closed-form rule, and the resulting vector of signed scores - the address - decides how strongly each of several small expert modules speaks, with negative scores subtracting an expert's contribution. The program calls this family of mechanisms the aleph (after ℵ, the set-theory symbol for graded infinities), and the models built around it carry the name. Selection by graded, signed blending is the design signature; softmax routing, the standard rule that converts scores into positive sum-to-one votes, appears nowhere in the family.

On top of each frozen trunk - the two encoders of section 1 - sits a small library of AMOE arms (anchored mixture-of-experts): adapters organized around anchor directions, blended by that graded signed router, each detachable to bit-exactness. Three arms (equivalence, simplification, paraphrase) lift the pair to .7287 and .7294, a +.12 jump over the bare trunks that closes most of the distance to the pair-trained reference model. The collective beats each of its own members on all eight tasks. The sharpest case is the medical-domain task: one arm, used alone there, actually scores below the bare trunk - yet the mixture including it beats everything. The capability lives in the blend, with each anchor contributing a perspective of its own (per-member tables in the companion).

Two findings from this library shaped the whole week. First, capacity control: a completely random adapter of the same size gains +.0045 by itself, so every candidate arm was judged against random-at-same-size - the control that prices the free lunch before any arm gets credit. Two plausible candidates (lexical and topical) measured below that bar and were rejected. Second, anchors are trunk-bound: transplanting one trunk's arms onto its sibling - the closest possible pair, same recipe and same data - costs 31% of their gain, and of that loss, 71% belongs to the anchors themselves and 29% to the small scoring layers beside them. Retraining the anchors natively recovers all of it. An adapter is a relationship with the specific trunk that grew it; unrelated trunks are untested and would likely cost more.

Why these points matter

• +.12 from 4.9M adapter parameters: the cheapest capability in the program, and it toggles off to a bit-identical baseline - the regression risk of adopting it is zero by construction.
• Random-arm control at +.0045: the free lunch, priced, before any adaptation claim gets credit.
• The 31% transplant cost, 71% of it in the anchors: portability is the exception; native fit is the rule.

3. One harness, ten models

Every capability number in this article comes from one benchmark harness that treats all ten models identically - the five teachers, the two trunks, both adapted collectives, and the pair-trained reference. Eight semantic-similarity tasks (STS-B, SICK-R, STS12–16, BIOSSES); every model mean-pooled and length-normalized (pooling: how a sentence's many token vectors are collapsed into one - here, the simple average); geometry gauges recorded beside every score. The harness replicated its own historical record to four decimal places when re-run on fresh hardware. That reproducibility is what lets this week's new family - the AlephLM-0 trunks, read through the summary-token pooling they were trained with - join the same tables on the same tasks.

4. The anchored trunk: AlephLM-0

The week's central experiment asked whether the anchor mechanism belongs inside a trunk. Three encoder variants were trained under the identical consensus recipe at exact capacity parity (the repo labels them a1/a2/a3). A dense control - the standard undivided feed-forward block, no routing at all. An anchored trunk whose feed-forward capacity splits into an always-on expert plus three experts dispatched per-token by a signed address with learned anchors. And a frozen-anchor twin with its anchors fixed at random birth. The dispatch machinery costs +0.063% parameters, and the routed trunks are born as their own null hypothesis: expert outputs zero-initialized, so at step zero the dispatch contributes exactly nothing, bit for bit.

The endpoint verdict is a three-way tie - at seed 0 and again at seed 1 - and the tie is exactly what the design hoped to see. The structure costs nothing at the endpoint, while the measurements around the tie show it working:

*Consensus retrieval: how often a caption's embedding retrieves its own consensus target - the training-time gauge, distinct from the benchmark scale. Seed 0 shown; full corpus (31.9M rows × 4 epochs). All three variants replicated at seed 1 - anchored .6007, dense .6040, frozen-anchor .6047 - widening the tie to a six-run, two-seed band of .6007–.6047.

The routing is load-bearing. Switching the dispatched experts off costs .026–.052 of capability, gracefully - the silenced quarter-width trunks (.552–.577) still clear raw mean-pooled BERT (.529). The per-task decomposition says what the routing carries: mostly the hardest tasks, led by the medical-domain one (a 100-pair benchmark, so the noisiest gauge in the suite) with the harder semantic suites steady at −.02 to −.08 each. How much it carries varies with seed - larger at seed 1 in both routed variants (−.038 and −.052, against −.029 and −.026 at seed 0), a two-point pattern that earns a third seed before it earns a name. The full per-run tables live in the companion.

Most of what the routing does is subtract. The first signed instruments, applied to the frozen-anchor trunk, found negative dispatch weights as a first-class mode - the majority mode for most experts, at a negative fraction of .42–.74 each. Subtraction is a structure invisible to every magnitude-based gauge that came before.

The anchored trunk learns faster on the small bed. One paired run at one seed - 500k rows, identical data and seed for both variants - and every gauge favors the anchored form, with the gap closing only under more epochs:

Endpoint equality and path advantage coexist: convergence tells one story, the learning curve tells another. Whether the speed comes from the learned anchors or from expert partitioning as such awaits the frozen twin on this bed.

All three variants are now seed-replicated, each with its training fingerprint effectively reproduced (the anchored pair: retrieval .9975 vs .9980, cosine .8391 vs .8394, rank 98.6 on both). Eight separately-trained trunks, spanning every design variation of the week, land on nearly identical internal statistics (target cosine ~.84, effective rank ~99–103): the family converges to the same place no matter how it is built, which makes it reliable experimental material. The full two-seed tables live in the companion.

Why these points matter

• The tie at +0.063% params: anchored capacity organization is free at trunk scale - structure can be added for its mechanism benefits with the capability bill at zero.
• Toggle −.026 to −.052, graceful: the null-path design goal realized - routing that can be removed and priced, its cost concentrated in the hardest tasks.
• Inhibition as a first-class mode: the signed address expresses what softmax routing structurally cannot - negative weights - and the trunks use that capacity heavily. A softmax-routed control at parity is the obvious next arm, and has not yet been trained.
• .9055 vs .7790 sample efficiency: at half a million rows the anchored trunk leads the dense control on every gauge on identical data - where budgets are finite, the anchored form is the economical learner.

5. Conditioning, depth, and a measured repair

Text encoders also serve as the conditioning signal for image generators - the role CLIP plays in Stable Diffusion - and that consumer reads per-token states, so token-state quality decides fitness for the job. Image generators famously truncate or degrade long prompts: past some position the encoder stops binding words to their attributes, and the picture stops obeying the text. How deep binding survives is therefore the number that matters. A parallel line of work measured it directly with four preregistered probes, and the trunks passed: token recoverability - can the original token be identified from its state - reads .984, above the .966 of T5 (a standard encoder used for exactly this conditioning job); symmetric attribute binding - whether each word's state still ties to its own attributes, red to the correct cube - holds; and the adapter arms raise token-grid rank by 12–17%. A deeper spectrum instrument then sharpened the picture: state similarity survives to great depths, while binding ends near position 480. Depth claims in this program now require spectrum sampling and a semantic gauge - a rule that immediately corrected one of our own earlier reads.

The repair became its own artifact: a 5M-parameter depth-extension arm over the frozen trunk, distilled from a long-context teacher (copied position patterns plus small trained adapters; the full recipe is in the companion). It passed its preregistered gate - binding at or above 0.8× the shallow-depth read at every measured depth through 2288. The copied initialization alone, zero parameters trained, restored binding at aligned depths, with dead edges every 256 positions that the trained arm then healed in a measurable near-to-far wave. The engaged mode carries a capability cost on short inputs (.038 on the 8-task mean, printed on the public model card beside the gains); disengaged, the model is bit-identical stock. Long-prompt conditioning, once a hard constraint, is now a switchable mode.

Why these points matter

• .984 recoverability from mean-pool training: pooled objectives still force per-token distinctness - the fear that token states were unsupervised byproducts is retired by measurement.
• Binding ends at 480 beneath a similarity plateau: gauges must match the consumer; similarity flatters, binding decides.
• A gate-passing depth arm at 5M params: depth is purchasable as an adapter, with the trunk untouched and the cost printed.

6. The first language-model rung

The adoption experiment attached the aleph's relay adapters - small insertable modules, each with a learned gate controlling how much it contributes at its position - to a frozen Qwen2.5-0.5B, distilling from its 1.5B sibling. Raw distillation scores tied against a plain-MLP control, and the mechanisms told fully distinct stories. The aleph adapter chose its own placement: it opened the gate at its output-side position eightfold - the position where the address directly shapes which word the model produces next - while the control closed the gate at that same position tenfold and worked from early layers instead. This rediscovers, by gradient, a placement rule the program had already found by measurement: adapt where the output distribution is shaped. Toggle discipline held at LLM scale to four decimals.

The task-transfer verdict was honestly null. Distilling on encyclopedic text moved both adapters closer to the teacher's outputs while costing both the same amount of task accuracy - an identical trade in both arms, which points at the corpus rather than either mechanism, and leaves the mechanisms untested for transfer until the next revision (instruction-domain data, already banked). A companion instrument settled a question that gates any multi-teacher LLM future: token-level alignment across different tokenizers is bridgeable. A learned map from one vocabulary to another picks the right counterpart token 84% of the time from a ~15,000-token lineup, against a same-tokenizer ceiling of .966 and shuffled-map floors of ~.0005. Tokenizer mismatch costs about half the alignment strength, and half survives - consensus methods can cross vocabulary boundaries.

7. The address itself: splat attention

The deepest work of the week returned to the core mechanism with a question posed as an image: many tiny codebooks watching the same data like observers of an inkblot - a shared shared sight, selection by opinion. An eight-probe mathematical battery ran before any architecture was built, and its verdicts now read as laws:

• The product code is real. At a fixed total budget of anchor directions, many small differentiated codebooks jointly address better than one large one - held-out readback rises monotonically from .859 to .955 as one codebook becomes sixteen.
• Differentiation is an attractor. Near-identical codebooks re-differentiate under a shared objective, because redundant opinions are gradient-wasteful. The data manufactures the perspectives.
• Composition rewards amplitude. Blending all perspectives in proportion to their signal beats selecting the best one, by a measured ~.10 - the program's oldest routing law, reproduced at a new level.
• Storage is partition-blind while addressing is partition-loving: how many facts the cells can hold scales with total cells alone; how finely they can be addressed scales with how the cells are divided. Two budgets, sized separately.
• Partition plus locality rescues what a shared store drowns. At equal total cells, 2048 tiny sharded heads retrieve at .9995 top-1 over a 2k context where a single monolithic store reads .042 - a 24× margin on that gauge and ~600× on the harder one (.934 vs .0015) - at cost linear in sequence length.

The attention block built from these numbers (softmax-free, signed addresses, write and read through shared cells) then handed the week its one real setback. Every static probe passed; the first training runs collapsed. Thirty-minute screens on a local card, run in place of sixty-hour rented runs, isolated the cause across three rounds of paired experiments: routing that owns no parameters forces the token states to do the routing's work, and they deform into address basins. Each place the routing was granted parameters of its own - a rotary position encoding inside the address, a learned address frame, codebooks that train - multiplied recovery by roughly 3×, and the three together converge. The collapse is arrested and the recipe converges: the block trains to 90% of a standard block's training gauge at matched small budget, still climbing at cutoff, though it has not yet faced a capability benchmark. The remaining gap is priced plainly at about 3× the optimization steps for the same ground at this scale. The public design record was amended accordingly: frozen addressing suffices for static mathematics; training requires routing-owned parameters. The prototype ships as a single self-contained file with the measured numbers, the failure modes, and the converging recipe printed in its header, at aleph-splat-0.

Why these points matter

• .859 → .955 at equal budget: perspective multiplicity is a real resource with a measured exchange rate.
• The 2048-head table (.9995 vs .042): the scaling shape for address-based attention - thousands of tiny heads are an engineering statement, at linear cost.
• The collapse-and-cure arc: static validity and trainability are separate properties; the program now tests both, cheaply, before any large run.
• ~3× optimization distance, stated plainly: the current price of softmax-free attention, and the target of the next round of schedule work.

8. The week's laws, in brief

A selection from the full law index. ("Law" is this program's internal term: a rule that has survived every paired test run so far, scoped to the beds that earned it - a claim about these models and this data, not about the field. The companion documents carry each law with its earning measurement.)

1. Agreement can exceed every source - given scale; the effect has a floor in joint model-and-data scale, now bracketed.
2. Capability ceilings are geometric budgets - teacher-agreement rank bounds the student, measurably, before training.
3. Frame choice decided deployment both times it was measured - the reference-aligned family shipped plain; the mean-aligned family shipped a rotation. One instance each: a design expectation, not yet a settled rule.
4. Adaptation is priced against random-at-same-size - the capacity control is the baseline.
5. Anchors are trunk-bound - the one sibling transplant measured cost 31%, most of it in the anchors' fit to their trunk; unrelated trunks are untested.
6. Graded blending beats hard selection - measured at ~.10 in codebook ensembles this week, consistent with the adapter and trunk results of prior installments.
7. Inhibition is a first-class mode - signed routing subtracts nearly as often as it adds, and magnitude gauges miss it entirely.
8. Endpoint convergence and path advantage coexist - ties at convergence, anchored efficiency along the way.
9. Depth claims require spectrum sampling and a semantic gauge - similarity plateaus flatter; binding decides.
10. Placement is elected at the output distribution - the mechanism's own gradients chose it at LLM scale; whether it pays in capability awaits the instruction-domain revision.
11. Many small differentiated codebooks out-address one large one at equal budget - and the differentiation maintains itself.
12. Static validity and trainability are separate properties - routing must own parameters to train.

9. What the week established

The arbitration that matters most is methodological. This week's system - consensus targets computed once and reused everywhere; frozen trunks with detachable, priced adapters; preregistered gates with falsification controls; small paired screens run before any large commitment; every artifact shipped with its numbers - reliably converts GPU-days into laws. The screens deserve the emphasis: one thirty-minute local experiment parked roughly sixty hours of queued rented compute, and screens then arbitrated three successive repair rounds of the attention prototype for the price of an afternoon. The shipped results range from a .7294 encoder built on agreement and anchors to a softmax-free attention block brought from collapse to convergence, and the cost per law has fallen all week.

Wall-clock per law is now the binding constraint, and the next phase is shaped accordingly - smaller batteries, bulk analysis. The open question that motivates all of it: carrying the anchored mechanism onto raw token streams with the same discipline.

10. Artifacts

11. Citations and attribution

Infrastructure. Training ran on RunPod (A40 and consumer-class rentals) and Google Colab; local development on consumer NVIDIA hardware. Built on PyTorch, the Hugging Face hub, transformers, datasets, and tokenizers, and lm-evaluation-harness for LLM task scores.

Teacher and comparator models. BERT (Devlin et al.), ModernBERT (Warner et al.), RoBERTa (Liu et al.), ALBERT (Lan et al.), DistilBERT (Sanh et al.), MiniLM (Wang et al.) via Sentence-BERT (Reimers & Gurevych), Longformer (Beltagy et al.), T5 (Raffel et al.), CLIP (Radford et al.), and Qwen2.5 (Qwen team).

Data. Conceptual Captions 12M (Changpinyo et al.) via the pixparse/cc12m-wds packaging, with LLaVA-style recaptioning; the MTEB similarity suites (STS12–16, STS-B, SICK-R, BIOSSES); WikiText; and the lm-eval task set (ARC-Easy, HellaSwag, PIQA, WinoGrande).

Methods. InfoNCE contrastive learning (van den Oord et al.); orthogonal Procrustes alignment (Schönemann, 1966); rotary position embeddings (RoFormer, Su et al.); zero-initialized adapter heads in the lineage of LoRA (Hu et al.); gradient checkpointing (Chen et al.); and mixture-of-experts routing in the lineage of Shazeer et al., departing from it by graded signed addressing throughout. The name splat nods to the picture of overlapping local supports composing a global field, as in 3D Gaussian Splatting (Kerbl et al.).

Companion technical documents accompany each model family with complete training configurations, per-task tables, geometry gauges, and instrument records.