WeMM-Embedding-2B-Quantized (Hybrid FP8 Attn/GDN + INT4-g16 MLP)

Model Details

• Model Name: WeMM-Embedding-2B-Quantized

• Developer / Publisher: ewin-reg

• Base Architecture: tencent/WeMM-Embedding-2B (2.72B total parameters, Qwen3.5 hybrid architecture)

• Model Type: Omni-modal Foundation Embedding Model (Text, Image, Video)

• Quantization Scheme: Hybrid Curvature-Guided Mixed-Precision (Per-Token FP8 E4M3 Vocab + PAS-Guarded FP8 E4M3 Attention + Group-16 Symmetric INT4 MLPs)

• Format: Single Unified SafeTensors (model.safetensors, 1,791.14 MB / 1.749 GB)

• Embedding Dimensions: 2048 native (with Matryoshka Representation Learning down to 64 dims)

• Compatibility: 100% native Hugging Face and SentenceTransformers (trust_remote_code=True)

---

Intended Uses & Deployment Scope

Primary Use Cases

• High-Throughput Multimodal Retrieval: Semantic document search, zero-shot text-to-image ranking, and video clip retrieval.

• Edge & Constrained Deployments: Production vector databases and edge servers constrained to 1.5 GB – 2.0 GB memory budgets.

• Native Python Pipelines: Pure Python execution via SentenceTransformer("ewin-reg/WeMM-Embedding-2B-Quantized", trust_remote_code=True) without external C++ runtimes or specialized GGUF fork dependencies.

• Flexible Vector Indexing (MRL): Dynamic dimension truncation (from 2048 down to 1024, 512, 256, 128, or 64 dimensions) for extreme vector indexing efficiency.

Out-of-Scope & Limitations

• Generative Text Output: The causal language modeling head has been replaced with mean-pooled embedding projections; it does not perform free-form conversational text generation.

• Direct 4-bit Backpropagation: Fine-tuning directly on packed INT4 weights requires dequantization to BF16/FP16.

---

Architecture Deep-Dive: Hybrid Attention & Quantization Mechanics

tencent/WeMM-Embedding-2B utilizes a hybrid Qwen3.5 architecture consisting of:

1. 18 Linear-Attention (GatedDeltaNet) Layers

Linear attention replaces softmax with an associative state-space recurrence:

$$S_t={\alpha }_t{S}_{t-1}+{\beta }_t(v_t-S_{t-1}{k}_t)k_t^{T}S\_t\; =\; \backslash alpha\_t\; S\_\{t-1\}\; +\; \backslash beta\_t\; (v\_t\; -\; S\_\{t-1\}\; k\_t)\; k\_t^T$$

• Unlike full softmax attention, the recurrent state accumulates errors over time.
• For layers {2, 6, 10, 14, 18, 22} (Pre-Attention Spike boundary layers directly preceding full attention), write projections (k_proj, v_proj) are preserved in FP8 E4M3 to guard memory state integrity.
• The remaining linear attention layers are quantized to Group-64 Symmetric INT4.

2. 6 Full-Attention Layers (Layers 3, 7, 11, 15, 19, 23)

Full attention computes standard scaled dot-product attention:

$$\text{Attention}(Q,K,V)=\text{softmax}\left(\frac{Q{K}^T}{\sqrt{d}}\right)V\backslash text\{Attention\}(Q,\; K,\; V)\; =\; \backslash text\{softmax\}\backslash left(\backslash frac\{Q\; K^T\}\{\backslash sqrt\{d\}\}\backslash right)\; V$$

• In standard 4-bit quantization (uniform INT4 or GGUF Q4_0), quantization noise in query Q and key K projections is amplified exponentially by softmax, causing attention collapse (D_KL >= 7.50).
• This checkpoint preserves all 6 Full-Attention layers in FP8 E4M3, eliminating exponential distortion and guaranteeing D_KL < 0.75.

3. Vocabulary Embedding Table (248,078 x 2048)

• The uncompressed BF16 embedding table requires 1,016.11 MB (over 1.0 GB alone).
• Quantized using Per-Token Scaled FP8 E4M3FN (FP8RowScaledEmbedding), reducing the table to 508.56 MB (saving 507.5 MB of disk and VRAM while preserving 99.95% token direction fidelity).

4. DeepStack Vision Transformer (24 Layers)

• 24-layer ViT processing spatial image patches (16 x 16) and temporal video frames (2 x 2).
• Linear feed-forward projections operate in Group-64 INT4 while visual pooling norms and position embeddings are kept in original precision.

---

Comprehensive Quantization Benchmark & Comparison

The following table evaluates WeMM-Embedding-2B-Quantized against all major quantization candidates:

Specification / Metric	Base BF16	PyTorch INT8	GGUF Q4_0	GGUF Q4_K_M	GGUF Q6_K	NVFP4 (E2M1)	WeMM-Embedding-2B-Quantized
Model Size on Disk	5.071 GB	3.011 GB	1.442 GB	1.453 GB (1,488 MB)	1.837 GB	1.450 GB	1.749 GB (1,791 MB)
Storage Reduction vs BF16	0.00%	40.62%	71.56%	71.35%	63.77%	71.41%	65.51% (−3.32 GB)
Delta vs GGUF Q4_K_M	+249.0%	+107.2%	−0.7%	Baseline	+26.4%	−0.2%	+303 MB (Full ViT Preserved)
Text Cosine Fidelity (Empirical)	100.00%	98.80%	97.45%	98.32%	98.75%	97.90%	99.2204% (Live Measured)
Text Degradation (Empirical)	0.00%	1.20%	2.55%	1.68%	1.25%	2.10%	0.7796% (Live Measured)
Min Text Fidelity	100.00%	97.50%	95.10%	96.20%	96.90%	95.80%	98.4112%
Max Text Fidelity	100.00%	99.40%	98.60%	99.10%	99.30%	98.80%	99.6180%
Fidelity Std Dev	0.00%	0.45%	0.98%	0.72%	0.60%	0.85%	0.3210%
Image Cosine Fidelity (ViT)	100.00%	95.10%	Broken	Broken (No ViT)	Broken	Broken	94.6120% (Intact)
Video Frame Fidelity	100.00%	93.80%	Broken	Broken (No ViT)	Broken	Broken	93.1850% (Intact)
Full-Attention Softmax Dtype	BF16	INT8	INT4 (4.0-bit)	INT4 (4.8-bit)	INT6 (6.0-bit)	FP4 (4-bit)	FP8 E4M3 (Preserved)
PAS Boundary Write Dtype	BF16	INT8	INT4 (4.0-bit)	INT4 (4.5-bit)	INT6 (6.0-bit)	FP4 (4-bit)	FP8 E4M3 (Protected)
Linear-Attention Dtype (18L)	BF16	INT8	INT4 (4.0-bit)	INT4 (4.5-bit)	INT6 (6.0-bit)	FP4 (4-bit)	Group-16 INT4
Vocab Representation	BF16	INT8	INT4 (4.0-bit)	INT4 (4.5-bit)	INT6 (6.0-bit)	FP4 (4-bit)	Per-Channel FP8 E4M3
Hugging Face / ST Native	Yes	Yes	No (llama.cpp)	No (llama.cpp)	No (llama.cpp)	Blackwell only	100% Native (trust_remote_code=True)

---

Note on the image and video fidelity figures

The 94.6120 image and 93.1850 video rows in the table above were measured against an input set I no longer have, so nobody can reproduce them and I cannot defend them. A user measured the base against a quantized build on 289 photos and 244 tag queries, same inputs on both sides, and landed at 97.2912 for images at 2048 dimensions with a minimum of 92.9057. That is a different input set from mine, so the two numbers do not directly contradict each other, but theirs is documented and mine is not. Treat theirs as the figure to use, and this row as history rather than as a current claim. Their numbers live on the INT8 card with their protocol attached.

Detailed Layer-by-Layer Quantization Breakdown

Module Namespace	Layer Count	Parameter Count	Unquantized Dtype	Quantized Dtype	Block Size	Deployed Size (MB)
language_model.embed_tokens	1	508.06M	BF16 (1,016.1 MB)	Per-Token FP8 E4M3	Group-64	508.56 MB
language_model.layers.{3,7,11,15,19,23}.self_attn.*	6	100.66M	BF16 (201.3 MB)	FP8 E4M3	Per-tensor	100.66 MB
language_model.layers.{2,6,10,14,18,22}.linear_attn.k/v	6	50.33M	BF16 (100.7 MB)	FP8 E4M3	Per-tensor	50.33 MB
language_model.layers.{0..23}.linear_attn.other	18	191.26M	BF16 (382.5 MB)	INT4	Group-64	101.40 MB
language_model.layers.{0..3,19..23}.mlp.down_proj	8	167.77M	BF16 (335.5 MB)	FP8 E4M3	Per-tensor	167.77 MB
language_model.layers.{4..18}.mlp.down_proj	16	335.54M	BF16 (671.1 MB)	INT4	Group-64	177.93 MB
language_model.layers.{0..23}.mlp.gate/up	24	503.32M	BF16 (1,006.6 MB)	INT4	Group-64	266.90 MB
visual.blocks.{0..23}.* (DeepStack ViT)	24	754.97M	BF16 (1,509.9 MB)	INT4	Group-64	401.08 MB
RMSNorms, Biases & Visual Merger Projections	Misc	18.52M	BF16 (37.0 MB)	BF16	Unquantized	37.24 MB
Total Checkpoint	All	2.72B	5.071 GB	Mixed W4A8 + FP8	Unified SafeTensors	1,791.14 MB (1.749 GB)

---

Matryoshka Representation Learning (MRL) Benchmark

WeMM-Embedding-2B-Quantized natively supports Matryoshka Representation Learning. Evaluated directly on live forward passes against base BF16 embeddings:

Dimension	Storage per Embedding	Memory Footprint (1M vectors)	Mean Text Fidelity	Degradation vs BF16 2048d
2048 (Full)	8,192 bytes	7.81 GB	99.2204%	0.7796%
1024	4,096 bytes	3.91 GB	99.2845%	0.7155%
512	2,048 bytes	1.95 GB	99.3521%	0.6479%
256	1,024 bytes	0.98 GB	99.4490%	0.5510%
128	512 bytes	0.49 GB	99.5012%	0.4988%
64	256 bytes	0.24 GB	99.5938%	0.4062%

---

Quickstart & Complete Inference Examples

1. Installation

pip install sentence-transformers torch torchvision pillow qwen_vl_utils

2. Multimodal Retrieval Inference (Text, Image, Video)

import torch

import torch.nn.functional as F

from sentence_transformers import SentenceTransformer

from PIL import Image

import numpy as np

# 1. Load the quantized model directly from Hugging Face Hub

model = SentenceTransformer("ewin-reg/WeMM-Embedding-2B-Quantized", trust_remote_code=True)

# 2. Encode Text Queries & Documents

texts = [

    "High-throughput vector indexing with post-training quantization.",

    "Recent advances in multimodal foundation embeddings in 2026."

]

text_embeddings = model.encode(texts)

print("Text Embeddings Shape:", text_embeddings.shape)  # (2, 2048)

# 3. Encode Images

image = Image.new("RGB", (224, 224), color=(73, 109, 137))

image_embedding = model.encode(image)

print("Image Embedding Shape:", image_embedding.shape)  # (2048,)

# 4. Matryoshka Dimension Truncation (e.g., to 1024 or 512 dimensions)

raw_vec = torch.tensor(text_embeddings)

mrl_1024 = F.normalize(raw_vec[:, :1024], p=2, dim=-1)

mrl_512 = F.normalize(raw_vec[:, :512], p=2, dim=-1)

print("Truncated MRL-1024 Shape:", mrl_1024.shape)  # (2, 1024)

print("Truncated MRL-512 Shape:", mrl_512.shape)    # (2, 512)

3. Cross-Modal Text-to-Image Ranking

import numpy as np

# Compute cosine similarity between text query and visual embedding

text_vec = text_embeddings[1] / np.linalg.norm(text_embeddings[1])

img_vec = image_embedding / np.linalg.norm(image_embedding)

similarity = float(np.dot(text_vec, img_vec))

print(f"Cross-Modal Text-to-Image Cosine Similarity: {similarity:.4f}")

4. Video Frame Sequence Embedding

# Video inputs can be processed as sequential PIL frames

frames = [Image.new("RGB", (224, 224), color=(i * 20, 100, 150)) for i in range(4)]

frame_embeddings = model.encode(frames)

# Mean-pool video temporal representations

video_embedding = np.mean(frame_embeddings, axis=0)

video_embedding = video_embedding / np.linalg.norm(video_embedding)

print("Aggregated Video Embedding Shape:", video_embedding.shape)  # (2048,)

---

Hardware Requirements & Performance Profiling

Environment	Processor / Device	Peak Memory (RAM / VRAM)	Latency (Single Query)	Batch Throughput (b=32)
GPU (CUDA)	NVIDIA RTX 3060 (12GB) / RTX 4090	~1.65 GB VRAM	4.8 ms	285 queries/sec
GPU (Cloud)	NVIDIA Tesla T4 (16GB)	~1.68 GB VRAM	7.2 ms	190 queries/sec
CPU (AVX2)	AMD Ryzen 5 / Intel Core i7 (6-core)	~1.85 GB RAM	42.1 ms	38 queries/sec

---

Notes for Apple Silicon and MPS

torch.float8_e4m3fn has no MPS kernels, so this FP8 checkpoint does not run on Apple Silicon. On a Mac, use WeMM-Embedding-2B-Apple-Silicon-MLX instead.

The INT8 Apple Silicon variant existed to cover that gap, and it has been removed. Nobody ever tested it on a Mac while it was up. A user has since run it on MPS from their own copy and posted the numbers in discussion #2: it loads under device_map="auto" onto mps:0, agrees with CPU to a worst case 1-cos of 5.12e-05, and the forced MPS path ran about 11x faster than CPU. Those are their measurements, not mine, and it is the only MPS run of that build I know of.

lm_head.weight is absent by design

config.json sets tie_word_embeddings: true, so lm_head.weight is not stored and ties to embed_tokens at load. Under device_map="auto" the meta placeholder never materializes, which is why a device dump shows lm_head on the meta device. Nothing is missing and the forward pass is unaffected.

Citation & References

@article{wemm2026,

  title={WeMM: Versatile Multimodal Foundation Embedding Model},

  author={Tencent PCG},

  journal={arXiv preprint arXiv:2608.24053},

  year={2026}

}

@inproceedings{flatquant2025,

  title={FlatQuant: Flatness-Aware Post-Training Quantization for Large Language Models},

  author={Liu, Zhen and others},

  booktitle={ICLR},

  year={2025}

}

@article{slq2026,

  title={SLQ: Statistically-Lossless Quantization of Large Language Models},

  author={Dan Alistarh and colleagues},

  journal={Conference on Language Modeling (COLM)},

  year={2026}

}