MAGI-7B-MoE

MAGI-7B-MoE is an experimental sparse Mixture-of-Experts language model developed by NULLXES as part of the MAGI model family.

MAGI is a research program focused on sparse model architectures, expert routing, scalable training systems, and synthetic intelligence.

Architecture

MAGI-7B-MoE uses a decoder-only sparse Mixture-of-Experts architecture.

Current architecture configuration:

• Decoder-only Transformer
• Sparse Mixture-of-Experts
• 64 routed experts
• 1 shared expert
• Top-4 expert routing
• 3 dense layers
• 29 MoE layers

The complete architecture configuration included with this release is available in:

magi_7b_moe_v0.1.yaml

Release status

This repository contains an early MAGI architecture checkpoint.

The checkpoint was produced to validate:

• MAGI model construction
• CUDA/H200 execution
• forward and backward passes
• sparse expert routing
• router behavior
• optimization
• gradient flow
• checkpoint serialization
• end-to-end MAGI training infrastructure

This checkpoint is not a production-ready base model.

It uses the temporary MAGI bring-up tokenizer:

magi_bringup_8k_v0.1

It must not be represented as a checkpoint trained with the final MAGI_TOKENIZER_V1.

The internal artifact metadata is retained in PROBE_NOT_BASE.json for reproducibility and release provenance.

Initial training validation

Hardware:

• NVIDIA H200
• CUDA
• BF16 supported

Training configuration:

• Steps: 50
• Sequence length: 1,024
• Batch size: 1
• Tokens processed: 51,200
• Learning rate: 3e-4

Observed training result:

• Initial loss: 9.4176
• Final loss: 3.2435
• Loss delta: 6.1741
• Final dead experts: 0
• Final router entropy: 4.0884
• Final expert imbalance ratio: 2.9494
• Mean throughput: ~1,392.7 tokens/s

Training status:

OK

These values describe an architecture bring-up run and should not be interpreted as downstream model-quality benchmarks.

Checkpoint

Weights are distributed as:

model.safetensors

Optimizer state is intentionally not included in this public package.

MAGI

MAGI is developed by NULLXES.

The broader MAGI program explores scalable sparse neural architectures and training infrastructure for synthetic intelligence systems.

---

© 2026 NULLXES