nileshsarkar-ai/sae-encoders

SAE Encoders — codebase + experiment artifacts

Codebase, results, figures, and paper artifacts for the COLM 2026 work on Sparse Autoencoder scaling laws (Erdős AI Labs).

Headline figures

Toy-model loss-floor curves (exp1, main paper result)	Per-layer SAE metric heatmap (run_trial_2)

The model weights themselves live in four sister Hugging Face repos (this repo holds the code + small artifacts; weights total ~195 GB so they're split out):

Hugging Face repo	Contents	Distinguishing setting
🔗 pythia-410m-saes-x32-l1-adaptive	144 SAE checkpoints (24 layers × 6 snapshots)	L1 = 5e-4 adaptive, target L0 ≈ 150
🔗 pythia-410m-saes-x32-l1-3e-4-fixed	144 SAE checkpoints	L1 = 3e-4 fixed
🔗 pythia-410m-saes-x32-l1-8e-5-fixed	144 SAE checkpoints	L1 = 8e-5 fixed
🔗 erdos-forge-7b-qlora	Qwen2.5-Coder-7B QLoRA adapters (FORGE)	unrelated CAD-reasoning project

Everything in this repo is also mirrored on GitHub at github.com/nileshsarkar-ai/Erdos-AI-Labs.

Repository structure

Path	What it is
run_trial_2/	Trial 2 — first full Pythia-410M SAE training run with adaptive L1 (see run_trial_2/README.md)
run_trial_3/	Trial 3 — Trial-2 successor with dead-feature resampling, L1 warmup, ghost-grad fixes (see run_trial_3/README.md)
run_exp_2_t1/	Exp 2, run 1 — fixed L1 = 3e-4 sweep point (see run_exp_2_t1/README.md)
run_exp_2_t2/	Exp 2, run 2 — fixed L1 = 8e-5 sweep point + heatmap visualization suite (see run_exp_2_t2/README.md)
exp1/	Toy-model validation — KD minimum-width theorem (see exp1/README.md)
exp1_results_COLM/	Paper-ready figures + tables from exp1 (10 figures, 300 dpi PNG+PDF, JSON tables)
exp1_results_1k/	Earlier 1k-config sweep of exp1
older/	Archived earlier scripts and SAE runs (Pythia-110M / single-layer experiments)
ERDOS/	FORGE project — constraint-graph CAD-reasoning system (separate from SAE work)
colm-docs/	Paper PDFs (deep-dive, experiment plan, full paper draft)

Quick start

1. Pull the checkpoints for one of the runs:

   hf download nileshsarkar-ai/pythia-410m-saes-x32-l1-adaptive \
     --local-dir ./run_trial_2/checkpoints
   

2. Re-run the measurement phase without retraining:

   cd run_trial_2
   python run_trial_2.py --phase measure
   

3. Regenerate figures:

   python visualize_heatmap.py
   

Hardware used

NVIDIA A100 80 GB PCIe · Intel Xeon 42-core Cascade Lake · 62 GB RAM Each full training run (300 M tokens, 24 layers in parallel) took ~24 h.

Contact

For questions: Nilesh Sarkar — nileshsarkar.cs@gmail.com

Citation

If you build on this work, please cite the COLM 2026 paper (see colm-docs/).