| --- |
| license: other |
| tags: |
| - magi |
| - mixture-of-experts |
| - moe |
| - language-model |
| - transformer |
| - nullxes |
| --- |
| |
| # MAGI-7B-MoE |
|
|
| **MAGI-7B-MoE** is an experimental sparse Mixture-of-Experts language model developed by **NULLXES** as part of the MAGI model family. |
|
|
| MAGI is a research program focused on sparse model architectures, expert routing, scalable training systems, and synthetic intelligence. |
|
|
| ## Architecture |
|
|
| MAGI-7B-MoE uses a decoder-only sparse Mixture-of-Experts architecture. |
|
|
| Current architecture configuration: |
|
|
| - Decoder-only Transformer |
| - Sparse Mixture-of-Experts |
| - 64 routed experts |
| - 1 shared expert |
| - Top-4 expert routing |
| - 3 dense layers |
| - 29 MoE layers |
|
|
| The complete architecture configuration included with this release is available in: |
|
|
| `magi_7b_moe_v0.1.yaml` |
|
|
| ## Release status |
|
|
| This repository contains an **early MAGI architecture checkpoint**. |
|
|
| The checkpoint was produced to validate: |
|
|
| - MAGI model construction |
| - CUDA/H200 execution |
| - forward and backward passes |
| - sparse expert routing |
| - router behavior |
| - optimization |
| - gradient flow |
| - checkpoint serialization |
| - end-to-end MAGI training infrastructure |
|
|
| This checkpoint is **not a production-ready base model**. |
|
|
| It uses the temporary MAGI bring-up tokenizer: |
|
|
| `magi_bringup_8k_v0.1` |
|
|
| It must not be represented as a checkpoint trained with the final `MAGI_TOKENIZER_V1`. |
|
|
| The internal artifact metadata is retained in `PROBE_NOT_BASE.json` for reproducibility and release provenance. |
|
|
| ## Initial training validation |
|
|
| Hardware: |
|
|
| - NVIDIA H200 |
| - CUDA |
| - BF16 supported |
|
|
| Training configuration: |
|
|
| - Steps: 50 |
| - Sequence length: 1,024 |
| - Batch size: 1 |
| - Tokens processed: 51,200 |
| - Learning rate: 3e-4 |
|
|
| Observed training result: |
|
|
| - Initial loss: 9.4176 |
| - Final loss: 3.2435 |
| - Loss delta: 6.1741 |
| - Final dead experts: 0 |
| - Final router entropy: 4.0884 |
| - Final expert imbalance ratio: 2.9494 |
| - Mean throughput: ~1,392.7 tokens/s |
|
|
| Training status: |
|
|
| `OK` |
|
|
| These values describe an architecture bring-up run and should not be interpreted as downstream model-quality benchmarks. |
|
|
| ## Checkpoint |
|
|
| Weights are distributed as: |
|
|
| `model.safetensors` |
|
|
| Optimizer state is intentionally not included in this public package. |
|
|
| ## MAGI |
|
|
| MAGI is developed by NULLXES. |
|
|
| The broader MAGI program explores scalable sparse neural architectures and training infrastructure for synthetic intelligence systems. |
|
|
| --- |
|
|
| © 2026 NULLXES |
|
|
