mj0530/CERA-MoA-Hetero-3agents

CERA-MoA (Heterogeneous Pool, 3 Agents)

Official heterogeneous checkpoint for CERA-MoA: Co-Evolving Routing Mechanisms with Continually Learning LLM Agents.

Each agent has its own backbone (not a shared LoRA on one model). The router itself is trained on Qwen/Qwen3-4B.

• Paper: arXiv:2609.18779
• Code: github.com/michaeljiang0530/CERA-MoA
• Hugging Face Papers: hf.co/papers/2609.18779

What's included

Path	Description
agents/agent_0/	LoRA on Qwen/Qwen3-4B
agents/agent_1/	LoRA on microsoft/Phi-4-mini-instruct
agents/agent_2/	LoRA on meta-llama/Llama-3.2-3B-Instruct
router.pt	Familiarity predictor / router heads (Qwen3-4B)

Backbone weights are not included. Using agent 2 also requires the Llama 3.2 Community License.

Quick load (one agent)

from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "mj0530/CERA-MoA-Hetero-3agents"
bases = {
    0: "Qwen/Qwen3-4B",
    1: "microsoft/Phi-4-mini-instruct",
    2: "meta-llama/Llama-3.2-3B-Instruct",
}

i = 0  # 0=Qwen, 1=Phi, 2=Llama
base = bases[i]
tokenizer = AutoTokenizer.from_pretrained(base, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(base, torch_dtype="bfloat16", device_map="auto", trust_remote_code=True)
model = PeftModel.from_pretrained(model, f"{repo}/agents/agent_{i}")

For full multi-agent routing + inference, use the official code (train_gspo_hetero_agent.py / evaluation scripts) and point it to this folder (or a local download of the Hub repo).

Checkpoint note

This is the final snapshot from the paper's heterogeneous 3-agent run (Qwen3-4B + Phi-4-mini-instruct + Llama-3.2-3B-Instruct), corresponding to training step 6000.