AlephLM-0 — an anchored expert trunk, distilled against a dense control

This is a live experiment repository, not a finished model release. Runs land here as they finish training, checkpoints push every 30 minutes mid-run, and every arm ships — including any that end up refuted. If you are reading this while the run table below says IN PROGRESS, you are watching the experiment happen.

📄 Technical companion: TECHNICAL.md — the complete two-seed verdict, toggle decomposition, basin tables, instruments, and laws. Execution provenance (versions, container quotas, per-arm code revision): ENVIRONMENT.md. Week-in-review article: Geometric Memory FT5 — Agreement, Anchors, Addresses.

The question

Mixture-of-experts models normally route with a learned softmax over expert logits — a comparative choice among experts. This program tests a different router: a closed-form signed address over unit anchor directions,

u_k = cos(x, a_k) / τ          w_k = sinh(u_k) / Σ_j cosh(u_j)

where each expert's contribution is w_k · σ(g_k) · E_k(x) per token. The weights are signed — an expert can be recruited negatively (an inhibitory anchor) — and the read is reconstructive rather than competitive: no argmax, no top-k, no load-balancing loss. The anchors, gates, and experts are trained by nothing but the task gradient.

E1 (this repo): does a trunk built this way match or beat a parameter-matched dense trunk under an identical objective, at 32M-row scale? Six runs answer it:

run	encoder	routing	seeds
a1_anchored	trunk-expert ff512 + 3 dispatched experts ff512/block	signed aleph address, learned anchors	s0, s1
a2_dense	standard dense ff2048	— (the control)	s0, s1
a3_random	same as a1	anchors frozen at random init	s0, s1

a1 vs a2 is the headline; a1 vs a3 isolates whether learned addressing matters or any fixed partition of the capacity would do.

Results at a glance

All capability numbers are 8-task mean Spearman (STS-B, SICK-R, STS12–16, BIOSSES) from the certified harness in eval/alephlm0_benchmark.json; all training-side numbers are each run's checkpoints/metrics.json finals.

Endpoint — full corpus (31.9M rows × 4 epochs, CLS readout)

run	8-task mean	dispatch OFF	toggle	mimicry R@1	cos→target	erank /768
a1_anchored-s0	.6031	.5743	−.0288	.9980	.8394	98.6
a1_anchored-s1	.6007	.5625	−.0382	.9975	.8391	98.6
a2_dense-s0	.6026	—	—	.9975	.8418	99.2
a3_random-s0	.6033	.5772	−.0261	.9980	.8392	98.8
a2_dense-s1	.6040	—	—	.9975	.8420	99.2
a3_random-s1	.6047	.5523	−.0524	.9980	.8396	98.6

Three routing designs, one capability band (.0008 wide at seed 0, .0040 across all six runs), one training-gauge basin — at exact capacity parity, the dispatch machinery (+0.063% params) is free at the endpoint and carries .026–.052 of function by the toggle gauge.

Per-task Spearman — the tie in detail, plus the seed-1 replications

run	STS-B	SICK-R	STS12	STS13	STS14	STS15	STS16	BIOSSES	mean
a1_anchored-s0	.5731	.6528	.4996	.6014	.5457	.7121	.6761	.5639	.6031
a2_dense-s0	.5731	.6507	.4965	.5962	.5430	.7130	.6798	.5683	.6026
a3_random-s0	.5707	.6538	.4966	.5998	.5448	.7103	.6806	.5703	.6033
a1_anchored-s1	.5681	.6512	.4931	.5996	.5423	.7094	.6745	.5673	.6007
a2_dense-s1	.5740	.6537	.4997	.6038	.5417	.7128	.6789	.5671	.6040
a3_random-s1	.5717	.6533	.4946	.6007	.5434	.7128	.6800	.5809	.6047

What the dispatched experts carry — toggle decomposition (ON − OFF, per task)

run	STS-B	SICK-R	STS12	STS13	STS14	STS15	STS16	BIOSSES	mean Δ
a1_anchored-s0	−.008	+.030	−.020	+.029	−.010	+.035	+.053	+.121	+.0288
a1_anchored-s1	+.005	+.050	−.004	+.064	+.001	+.069	+.085	+.034	+.0382
a3_random-s0	−.006	+.023	+.002	+.026	−.004	+.025	+.050	+.093	+.0261
a3_random-s1	+.026	+.053	+.040	+.026	+.000	+.055	+.083	+.135	+.0524

Positive = function the routed experts carry; negative = the quarter-width trunk alone does slightly better there. In three of the four routed runs the largest single carrier is BIOSSES (+.09 to +.14); the anchored seed-1 run is the exception, shifting that weight onto the semantic suites (SICK-R, STS13/15/16). Both routed arms carry more total function at seed 1 (−.038 and −.052 vs −.029 and −.026) — where the routing concentrates varies by arm and seed; that it owns the hard semantics does not.

Sample efficiency — small bed (500k rows, L=128, single consumer GPU)

gauge	anchored	dense	gap
retrieval R@1, 2 epochs	.9055	.7790	+.1265
8-task capability, 2 epochs	.3833	.3530	+.0303
cos→target, 2 epochs	.6491	.5984	+.0508
effective rank, 2 epochs	46.1	35.7	+10.4
retrieval R@1, extended to 6 epochs	.9685	.9645	+.0040

Identical data, identical seed. Every gauge favors the anchored form at small budget; the dense control needs 3× the epochs to close the retrieval gap. Endpoint equality and path advantage coexist.

The basin — every completed full-scale trunk

run	mimicry R@1	cos→target	erank /768
a1_anchored-s0	.9980	.8394	98.6
a1_anchored-s1	.9975	.8391	98.6
a2_dense-s0	.9975	.8418	99.2
a2_dense-s1	.9975	.8420	99.2
a3_random-s0	.9980	.8392	98.8
a3_random-s1	.9980	.8396	98.6

The captionbert-v2/-B trunks land in the same attractor (cos→target ~.84, erank ~99–103): separately-trained trunks across two poolings, two corpus sizes, and three routing designs share one training-gauge basin.

Architecture

12 layers, d=512, 8 heads, pre-norm, 8192 learned positions, 768-d projected output, CLS readout (settled empirically — see S0e below).

• Per block, the dense FFN (ff2048) is replaced by 1 always-on trunk expert (ff512) + 3 dispatched experts (ff512 each) — 2048 hidden units total, exact capacity parity with the control.
• Dispatched-expert output layers are zero-initialized and gates start at σ(−3) ≈ 0.047: at initialization the dispatch contributes exactly zero (bit-exact, asserted at construction), so the anchored trunk is born as its own dense-trunk null hypothesis and the routing must earn its way in. One known consequence: the routing gradient is zero for exactly one step (∂L/∂w = σ(g)·E(x) and E ≡ 0 at init), the same dynamic as LoRA's A-matrix under B=0.
• Parameter cost of the machinery: +36,900 over dense (+0.063%) — 12 codebooks of 3×512, 36 gates, and the extra expert biases. 58,345,764 vs 58,308,864.

Training recipe (identical for every arm)

Consensus distillation, inherited verbatim from captionbert-8192-v2: the target for each caption is the L2-normalized centroid of five BERT-family teachers, each mapped into the reference member's frame (bert-base) by a whitened Procrustes fit — the precomputed targets cover ~32M captions (all 66 repaired chunks, 31,905,616 rows) from CC12M.

• loss = InfoNCE(T=0.07, in-batch negatives) + MSE (F.mse_loss, per-element mean — the batch of 2048 is the negative set, so batch size is part of the objective and is never changed)
• pure Adam (no weight decay), lr 6e-4, linear warmup 2000 → cosine to 1e-6, grad clip 1.0, AMP fp16, 4 epochs over 64 train chunks (31.9M rows), 2 holdout chunks for eval
• length-bucketed dynamic padding (ceiling 256 tokens), gradient checkpointing
• trained on a single RTX 5090 (32GB); worst-case batch measured 30.1 GB reserved

Stage-0 instruments (complete)

S0a — is the rank ceiling the teachers' agreement, or bert's own geometry? (s0a/s0a_erank.json) The consensus target occupies an effective rank of 28.1/768. Raw bert-base rows on the same corpus: 40.7/768 — and 40.3 on out-of-domain STS-B text, so the low rank is the encoder's geometry, not the corpus. Verdict at the matched (L2-normalized) gauge: ratio 1.45× → intermediate — the consensus construction costs ~30% of the member's rank, but the member itself only has ~40 directions to give. Any consensus built in a bert frame is capped near 40 regardless of teacher roster.

S0e — pooling settle (runs/alephlm0-s0e-*). Three identical dense trunks, one seed shared exactly (same init, same batch plan), differing only in readout, 500k rows × 2 epochs:

readout	cos→target	mimicry R@1
mean over mask	.6037	.7745
CLS token	.6147	.8180
learned-query attention	.6033	.7680

CLS wins both gauges, outside the preregistered tie band (.003 cos / .01 R@1) — notable because the target is a mean-pooled object, and the attention readout (initialized to be exactly mean pooling) declined to move away from mean. Stage 1 therefore trains with the CLS readout.

Run status

run	status
runs/alephlm0-s0e-{mean,cls,attn}	✅ complete
s0a/ erank instrument	✅ complete
runs/alephlm0-a2_dense-s0	✅ complete — mimicry R@1 .9975, cos→target .8418, erank 99.2/768; 8-task capability .6026 (eval/), inside the captionbert-v2/-B band: the dense recipe is four-times replicated at capability (.6026–.6077)
runs/alephlm0-a3_random-s0	✅ complete — mimicry .9980, cos→target .8392, erank 98.8; capability .6033 (band center: frozen-random routing matches dense at capacity parity); dispatch-OFF .5772 (toggle −.026) — the routed experts carry .026 of task function, degrading gracefully (eval/)
runs/alephlm0-a1_anchored-s0	✅ complete — mimicry .9980, cos→target .8394, erank 98.6; capability .6031, dispatch-OFF .5743 (toggle −.0288). Anchors moved 1.06 rad from init; amplitude .101
runs/alephlm0-a1_anchored-s1	✅ complete — capability .6007 (seed-stable vs s0's .6031), dispatch-OFF .5625 (toggle −.0382); mimicry .9975, cos→target .8391, erank 98.6 — the s0 fingerprint reproduced to three decimals
runs/alephlm0-a2_dense-s1	✅ complete — capability .6040 (seed pair .6026/.6040 brackets the band); mimicry .9975, cos→target .8420, erank 99.2 — the dense control is now seed-replicated in the basin
runs/alephlm0-a3_random-s1	✅ complete — capability .6047 (seed pair .6033/.6047), dispatch-OFF .5523 (toggle −.0524, the largest routed function measured); mimicry .9980, cos→target .8396, erank 98.6

E1 verdict, two seeds complete: the tie holds. Learned-anchor, frozen-anchor, and dense trunks land within .0008 of each other at seed 0 and within .0040 across all six runs — the anchored form costs nothing at exact capacity parity, and learned addressing lands even with a frozen random partition as an encoder trunk, even though it visibly reorganizes (anchors rotate a full radian, and every routed run carries real function by the toggle gauge, growing at seed 1 to −.052). The signed-address form's predicted advantage lives where the address parameterizes the output distribution — that is Stage 2's generative bed, which this result gates nothing about.

Each run directory carries checkpoints/ (state + rolling model snapshots + final_model.pt + metrics.json), config/ (the exact resolved configuration), and tensorboard/. Anchored runs additionally log per-block routing vitals at every eval: mean dispatched amplitude |w·σ(g)|, anchor drift from initialization, gate openings, and address-usage diversity — the curves that show the routing waking from its zero-initialized silence.

Lineage

• Teachers: bert-base-uncased, ModernBERT-base, roberta-base, albert-base-v2, distilbert-base-uncased (mean-pooled, 512-token truncation)
• Dense-recipe provenance: captionbert-8192-v2 (.6077 8-task STS mean, beating its best teacher at 13% of the combined teacher parameters) and its replication captionbert-8192-v2-B
• The signed-address form and its training laws come from a long-running research program on geometric routing (AMOE); the amplitude-conservation result that motivates per-token signed dispatch was established on adapter collectives before being carried inward here.

Maintained as a live research log. Numbers in this card are measured, not projected; anything not yet measured is marked as such.