---
license: other
tags:
  - magi
  - mixture-of-experts
  - moe
  - language-model
  - transformer
  - nullxes
---

# MAGI-7B-MoE

**MAGI-7B-MoE** is an experimental sparse Mixture-of-Experts language model developed by **NULLXES** as part of the MAGI model family.

MAGI is a research program focused on sparse model architectures, expert routing, scalable training systems, and synthetic intelligence.

## Architecture

MAGI-7B-MoE uses a decoder-only sparse Mixture-of-Experts architecture.

Current architecture configuration:

- Decoder-only Transformer
- Sparse Mixture-of-Experts
- 64 routed experts
- 1 shared expert
- Top-4 expert routing
- 3 dense layers
- 29 MoE layers

The complete architecture configuration included with this release is available in:

`magi_7b_moe_v0.1.yaml`

## Release status

This repository contains an **early MAGI architecture checkpoint**.

The checkpoint was produced to validate:

- MAGI model construction
- CUDA/H200 execution
- forward and backward passes
- sparse expert routing
- router behavior
- optimization
- gradient flow
- checkpoint serialization
- end-to-end MAGI training infrastructure

This checkpoint is **not a production-ready base model**.

It uses the temporary MAGI bring-up tokenizer:

`magi_bringup_8k_v0.1`

It must not be represented as a checkpoint trained with the final `MAGI_TOKENIZER_V1`.

The internal artifact metadata is retained in `PROBE_NOT_BASE.json` for reproducibility and release provenance.

## Initial training validation

Hardware:

- NVIDIA H200
- CUDA
- BF16 supported

Training configuration:

- Steps: 50
- Sequence length: 1,024
- Batch size: 1
- Tokens processed: 51,200
- Learning rate: 3e-4

Observed training result:

- Initial loss: 9.4176
- Final loss: 3.2435
- Loss delta: 6.1741
- Final dead experts: 0
- Final router entropy: 4.0884
- Final expert imbalance ratio: 2.9494
- Mean throughput: ~1,392.7 tokens/s

Training status:

`OK`

These values describe an architecture bring-up run and should not be interpreted as downstream model-quality benchmarks.

## Checkpoint

Weights are distributed as:

`model.safetensors`

Optimizer state is intentionally not included in this public package.

## MAGI

MAGI is developed by NULLXES.

The broader MAGI program explores scalable sparse neural architectures and training infrastructure for synthetic intelligence systems.

---

© 2026 NULLXES