Realmbird/Llama3_1-8B-Instruct-L19R-8x-ft

Llama Scope L19R-8x, continued-trained on Llama-3.1-8B-Instruct chat activations (F-BOS)

A 32,768-latent JumpReLU sparse autoencoder on the layer-19 residual stream (blocks.19.hook_resid_post) of meta-llama/Llama-3.1-8B-Instruct. It is Llama Scope's L19R-8x SAE (fnlp/Llama3_1-8B-Base-LXR-8x, trained on the base model), continued-trained for 19,996,672 tokens of WildChat conversations run through the Instruct model, then re-thresholded so mean L0 = 50. Latent indices are unchanged, so any per-latent analysis of the original Llama Scope dictionary (Neuronpedia llama3.1-8b/19-llamascope-res-32k, SelfIE labels, ...) can be tracked latent by latent.

Arm F-BOS: token 0 (BOS) was included in the training stream.

Author: Christopher Kinoshita (AI Alignment Foundation fellowship, p2-selfie). Uploaded by upload_hf.py.

Recipe

initialised from	fnlp/Llama3_1-8B-Base-LXR-8x · Llama3_1-8B-Base-L19R-8x (Llama Scope L19R-8x, sae_lens llama_scope_lxr_8x / l19r_8x)
activations	meta-llama/Llama-3.1-8B-Instruct (bf16), blocks.19.hook_resid_post = output of model.layers[19]
data	allenai/WildChat-1M, English, non-toxic, rendered with the Llama-3.1 chat template incl. the default system header, truncated to 1024 tokens
tokens seen	19,996,672 (4,882 steps of 4,096 tokens)
objective	TopK k=50 selected on act·‖W_dec‖ (Llama Scope's sparsity_include_decoder_norm), MSE reconstruction, no aux loss, decoder norms free
optimiser	Adam lr 0.0002, 5% warmup, 20% linear decay, seed 0
input scale	inputs multiplied by Llama Scope's own constant s = √4096 / 17.125 = 3.73723 during training; folded back into the weights here, so the SAE reads raw activations
token 0 (BOS)	included in the training stream
θ refit	after training, per-latent JumpReLU thresholds θᵢ = c / ‖W_dec,ᵢ‖ (checkpoint-space decoder norms) were re-fit on 500,000 held-out chat tokens (token 0 excluded) so mean L0 = 50: c = 1.79879 (Llama Scope's own c: 1.9708); mean L0 under the refit θᵢ 49.846, under sae_lens's flat 1.9708 28.929
final train loss / FVU	763.514 / 0.1101

Full record in sae_config.json.

Numbers, before and after

Metrics from eval_sae.py on Instruct activations, token 0 excluded from every per-token metric. B0 is Llama Scope as sae_lens loads it (flat θ = 1.971, L0 ≈ 34); B0-pub is the same weights under Llama Scope's intended per-latent gate (L0 ≈ 50); F-BOS is this checkpoint with its refit per-latent θ. See the threshold note below.

chat-eval (2M held-out WildChat tokens, chat template):

metric	B0 (sae_lens, flat θ = 1.971)	B0-pub (Llama Scope's rule, θᵢ = 1.971/‖W_dec,ᵢ‖)	F-BOS (this repo, refit θᵢ)
FVU	0.399	0.355	0.269
mean L0	33.308	48.660	50.491
ΔCE (nats)	0.389	0.237	0.180
CE recovered	0.960	0.975	0.981
n latents firing on token 0 in ≥90% of sequences	19,967	20,301	2,677
dead latents	2,097	933	881

text-eval (1M SlimPajama tokens, no template, through the Instruct model):

metric	B0 (sae_lens, flat θ = 1.971)	B0-pub (Llama Scope's rule, θᵢ = 1.971/‖W_dec,ᵢ‖)	F-BOS (this repo, refit θᵢ)
FVU	0.341	0.304	0.348
mean L0	34.274	50.498	57.819
ΔCE (nats)	0.339	0.244	0.355
CE recovered	0.963	0.973	0.961
n latents firing on token 0 in ≥90% of sequences	19,967	20,301	2,677
dead latents	819	362	488

Threshold note

sae_lens loads Llama Scope by folding the input scale s = √4096/17.125 = 3.737 into W_enc and multiplying the published JumpReLU threshold 0.52734375 by s as well, giving a flat θ = 1.971 on every latent; but the fold leaves b_enc alone, so the pre-activation is the same number in both spaces and the scaled θ is stricter than intended — the SAE runs at L0 ≈ 34 (34.2 on Llama-3.1-8B base + SlimPajama text, its own training distribution) instead of the advertised 50. Llama Scope's intended gate is pre · ‖W_dec,ᵢ‖ > 1.971 (sparsity_include_decoder_norm), i.e. a per-latent θᵢ = 1.971 / ‖W_dec,ᵢ‖ with checkpoint-space decoder norms, which gives L0 ≈ 50 (50.3 on the same data); the tables above call that operating point B0-pub and the sae_lens loading B0. Anyone loading fnlp/Llama3_1-8B-Base-LXR-8x through sae_lens hits the same issue. This checkpoint's saved threshold tensor is per-latent, θᵢ = c / ‖W_dec,ᵢ‖ with c = 1.79879 refit on 500,000 held-out chat tokens (token 0 excluded) so that mean L0 = 50 under the same rule, so pre > threshold — what the plain-torch snippet below and sae_lens's JumpReLUSAE.encode both compute — is the right rule for it.

What changed

• Chat improved, text got worse. Against B0-pub (same L0 ≈ 50): chat FVU 0.355 → 0.269, chat ΔCE 0.237 → 0.180 nats; text FVU 0.304 → 0.348, text ΔCE 0.244 → 0.355. The finetune specialises to the chat format; it does not learn a generally better dictionary for the Instruct residual stream.
• F-BOS includes token 0 (BOS) in the training stream. The number of latents that fire on token 0 in ≥ 90% of sequences collapses from 19,967 (61% of the dictionary) for B0 to 2,677 (8%), and dedicated BOS latents appear: 164 latents have ≥ 95% of their top activations at position 0 (B0: 16,485, i.e. the smear), a few dozen of which never fire at any later position (list in the AIAFC-69 write-up, H5).
• Decoder drift. Median decoder cosine to the original Llama Scope row 0.887 (mean 0.831, only 1.1% of rows above 0.99). Indices align but many rows moved, so per-latent labels from the original Llama Scope should be checked against dec_cos before reuse.

Files and how to load

file
sae.safetensors	W_enc [4096, 32768], W_dec [32768, 4096], b_enc [32768], b_dec [4096], threshold [32768]; float32, raw-activation space
sae_weights.safetensors + cfg.json	the same tensors in the sae_lens layout (architecture: jumprelu)
sae_config.json	training record written by finetune_sae.py

Plain torch (no sae_lens):

import torch
from safetensors.torch import load_file
from huggingface_hub import hf_hub_download

sd = load_file(hf_hub_download("Realmbird/Llama3_1-8B-Instruct-L19R-8x-ft", "sae.safetensors"))
W_enc, W_dec, b_enc, b_dec, threshold = (sd[k] for k in ("W_enc", "W_dec", "b_enc", "b_dec", "threshold"))

def encode(x):                       # x: [..., 4096] raw residual stream at blocks.19.hook_resid_post
    pre = x @ W_enc + b_enc
    return pre * (pre > threshold)   # JumpReLU; a latent "fires" iff its output is > 0

def decode(acts):
    return acts @ W_dec + b_dec

x is the output of model.layers[19] of meta-llama/Llama-3.1-8B-Instruct (HF transformers), i.e. TransformerLens blocks.19.hook_resid_post, in its raw scale. Do not rescale or normalise it: Llama Scope's dataset-norm factor is already folded into the weights.

sae_lens (≥ 6.x):

from huggingface_hub import snapshot_download
from sae_lens import SAE

path = snapshot_download("Realmbird/Llama3_1-8B-Instruct-L19R-8x-ft", allow_patterns=["cfg.json", "sae_weights.safetensors"])
sae = SAE.load_from_disk(path)        # -> JumpReLUSAE, d_in 4096, d_sae 32768
acts = sae.encode(x)                  # same result as the plain formula above

Caveats

• The decoder moved. Cosine similarity between each latent's decoder row here and in the original Llama Scope: median 0.8871, mean 0.8312, min -0.0078, fraction > 0.99: 0.011. Latent indices align with the original, but a latent's meaning may have drifted, more so for low-cos rows; check dec_cos before reusing an original label.
• Decoder rows are not unit-norm (Llama Scope trains them free): row norms 0.1626 – 0.8432, mean 0.4449. TopK during training selected on act·‖W_dec‖.
• Threshold. threshold is per-latent, θᵢ = c / ‖W_dec,ᵢ‖ with c = 1.79879 (range 0.5708 – 2.9601, median 1.0906), fit post hoc for mean L0 = 50 on chat text with token 0 excluded. It is Llama Scope's JumpReLU rule written out per latent, not a learned threshold; sae_lens loads it as JumpReLUSAE and ignores k. Under sae_lens's flat 1.971 this checkpoint would run at L0 ≈ 28.929.
• Token 0 / BOS. This arm saw BOS activations in training, so position 0 behaves differently from Llama Scope (which excluded BOS/EOS): token 0 fires 2,677 latents here vs 19,967 for B0, and the dictionary contains dedicated BOS latents. Still exclude token 0 from per-token statistics unless BOS is what you study.
• Inputs are meta-llama/Llama-3.1-8B-Instruct activations; Llama Scope's published WildChat transfer check was aggregate-only. Whether the finetune changed anything for a given latent is exactly what the numbers above and the AIAFC-69 write-up are about; treat this as a research artefact, not a drop-in replacement.

Provenance

AI Alignment Foundation fellowship, project p2-selfie, spike AIAFC-69 (finetune Llama Scope L19R-8x on Llama-3.1-8B-Instruct activations), follow-up to AIAFC-55/56 on always/never-activating SelfIE latents. Code: Christopher-Selfie/spikes/AIAFC-69-llamascope-instruct-finetune/ (finetune_sae.py, eval_sae.py, upload_hf.py; the threshold finding above comes from check_threshold.py and check_threshold_variants.py), git SHA cc0c46212d354315677302cbee1e4396677a2fb3, trained 2026-09-24T21:34:46.085408+00:00.

Citations

Llama Scope (the original SAE):

@article{he2024llamascope,
  title   = {Llama Scope: Extracting Millions of Features from Llama-3.1-8B with Sparse Autoencoders},
  author  = {He, Zhengfu and Shu, Wentao and Ge, Xuyang and Chen, Lingjie and Wang, Junxuan and Zhou, Yunhua and
             Liu, Frances and Guo, Qipeng and Huang, Xuanjing and Wu, Zuxuan and Jiang, Yu-Gang and Qiu, Xipeng},
  journal = {arXiv preprint arXiv:2410.20526},
  year    = {2024},
  url     = {https://arxiv.org/abs/2410.20526}
}

Base-to-chat SAE transfer and finetuning (the recipe this follows):

@misc{kissane2024saestransfer,
  title  = {SAEs (usually) Transfer Between Base and Chat Models},
  author = {Kissane, Connor and Krzyzanowski, Robert and Conmy, Arthur and Nanda, Neel},
  year   = {2024},
  howpublished = {AI Alignment Forum},
  url    = {https://www.alignmentforum.org/posts/fmwk6qxrpW8d4jvbd/saes-usually-transfer-between-base-and-chat-models}
}

Base model: Llama 3.1 (Meta), used under the Llama 3.1 Community License.