Sazanami-MetaEmbed-310m

English | 日本語

A Japanese retrieval model trained from sbintuitions/modernbert-ja-310m that simultaneously learns late-interaction retrieval and summarization into a small number of vectors. It summarizes queries into up to 16 vectors and documents into up to 64 vectors, with the number of vectors determined independently of the number of tokens in the text.

A feature of this model is that retrieval works even when only the leading part of the stored vectors is used. You can build an index with 16 query vectors and 64 document vectors, and choose to use only 8 query vectors and 32 document vectors during similarity computation. You can choose the trade-off between accuracy and computational cost without retraining.

1. Model Overview

Property	Value
Base model	sbintuitions/modernbert-ja-310m
Architecture	Transformer with summary tokens → token-level linear projection (768 → 128, no bias) → L2 normalization
Parameters	About 315M (61,440 for summary tokens, 98,304 for the projection layer)
Vector dimensions	128
Number of vectors per query	16 (fixed). Can be sliced to the first 1 / 2 / 4 / 8
Number of vectors per document	64 (fixed). Can be sliced to the first 1 / 4 / 8 / 16 / 32
Recommended input length	256 tokens + 16 summary tokens for queries / 1,024 tokens + 64 summary tokens for documents
Scoring function	MeanMaxSim (average over query tokens of the maximum similarity)
Prefix	16 summary tokens + 検索クエリ: ("search query: ") for queries, 64 summary tokens + 検索文書: ("search document: ") for documents
Main framework	sentence-transformers 6.0.0 or later (MultiVectorEncoder)
Mixed precision during training	bf16

Related Models

All four models were trained from sbintuitions/modernbert-ja-310m on the same training data. What differs is how many vectors represent a single document. Sazanami-Embed-310m is a single-vector model, trained under the same conditions for comparison.

Model	Vectors per document	When to choose
Sazanami-ColBERT-310m	Same as the number of non-padding tokens in the text	Highest accuracy when searching for long documents among many candidates
Sazanami-AGC-310m	Up to 32	When you want to reduce storage size
Sazanami-MetaEmbed-310m (this model)	Up to 64 (the number used can be reduced)	When you want faster similarity computation, or want to choose the number of vectors later
Sazanami-Embed-310m	1 (768 dimensions)	Single-vector. When you want to minimize storage size and computational cost

2. Underlying Research

Late Interaction

Late interaction is a retrieval method that encodes queries and documents separately and matches them token by token only during similarity computation. It was introduced in ColBERT (Khattab and Zaharia, 2020, Santhanam et al., 2022). While it preserves fine-grained matches, storing as many vectors as there are tokens for each document results in large storage sizes and long similarity computation times.

MetaEmbed

MetaEmbed is a method that directly generates a small number of vectors, rather than aggregating token representations of the text (MetaEmbed: Scaling Multimodal Retrieval at Test-Time with Flexible Late Interaction, arXiv:2509.18095). While the original paper targets multimodal retrieval, this model applies it only to Japanese text.

The mechanism is as follows:

1. Place summary tokens — Dedicated tokens are placed at the beginning of the input: 16 for queries and 64 for documents, each having separate parameters. The embeddings of these tokens do not expand the vocabulary table; they are provided as trainable parameters of the model.
2. Extract representations only at summary token positions — From the final layer, only the representations at the positions of the summary tokens are returned. Tokens of the text do not appear in the output. Projection and L2 normalization are subsequently applied.

Another pillar is Matryoshka Multi-Vector Retrieval. To enable retrieval across all slicing configurations—from 1 to 16 vectors and from 1 to 64 vectors—the loss is computed for each combination and averaged. This yields multiple accuracy and computational cost configurations from a single training run.

Training was conducted on six combinations of (number of query vectors, number of document vectors): (1, 1), (2, 4), (4, 8), (8, 16), (8, 32), and (16, 64). Users choose from among these six combinations. Slicing configurations other than these were not trained.

The score is MeanMaxSim. MaxSim is divided by the number of query vectors.

$$\text{MeanMaxSim}(q,d)=\frac{1}{\mathrm{\mid }q\mathrm{\mid }}\sum _i\underset{j}{\max }{q}_i\cdot d_j\backslash text\{MeanMaxSim\}(q,\; d)\; =\; \backslash frac\{1\}\{|q|\}\; \backslash sum\_\{i\}\; \backslash max\_\{j\}\; \backslash ;\; q\_i\; \backslash cdot\; d\_j$$

Because the number of query vectors varies depending on the combination, keeping the sum would result in inconsistent score scales. Dividing by the number of vectors aligns the scale across all combinations.

3. Usage

Load the model with MultiVectorEncoder in sentence-transformers 6.0.0 or later.

from sentence_transformers import MultiVectorEncoder

model = MultiVectorEncoder("mahiyama/Sazanami-MetaEmbed-310m", trust_remote_code=True)

# Queries: "Where can I get a certificate of residence?", "I want to know the enrollment procedure for National Health Insurance"
queries = [
    "住民票はどこで取れますか",
    "国民健康保険の加入手続きを知りたい",
]
# Documents: on obtaining a certificate of residence, National Health Insurance enrollment, and oversized garbage collection
documents = [
    "住民票の写しは、市区町村の窓口またはコンビニ交付サービスで取得できます。",
    "国民健康保険の加入手続きは、転入や退職から 14 日以内に市区町村の窓口で行ってください。",
    "粗大ごみの収集は、事前の申し込みが必要です。",
]

# Prefixes and summary tokens are stored in the model and do not need to be added to the text
query_vectors = model.encode_query(queries)        # 16 vectors per query
document_vectors = model.encode_document(documents)  # 64 vectors per document

scores = model.similarity(query_vectors, document_vectors)
print(scores)

To reduce the number of vectors, keep only the leading vectors of the encoded output.

# Compute similarity with 8 query vectors and 32 document vectors
scores = model.similarity(
    [v[:8] for v in query_vectors],
    [v[:32] for v in document_vectors],
)

4. Training

Base Model

The model is initialized with sbintuitions/modernbert-ja-310m. It has undergone only masked language modeling pre-training and has not been trained for retrieval. Both the projection layer required for late-interaction retrieval and the summary tokens used for summarization were newly initialized. The summary tokens start from values obtained by adding small random noise to the mean of the vocabulary embeddings.

In other words, this model carries out training for late interaction and summarization into a small number of vectors together in a single training run. This configuration achieved higher accuracy than initializing from an already trained late-interaction model.

Training Data

The same data as the original model was used. Two formats were used: n-tuples, pairing one query with one positive and five negatives, and pairs, pairing queries with only a positive.

Source	Format	Count
mahiyama/auto-wiki-qa	n-tuples	100,000
mahiyama/mqa-ja	n-tuples	100,000
mahiyama/mmarco-ja	n-tuples	50,000
mahiyama/civicqa-ja (non-public)	n-tuples	43,382
mahiyama/quiz-no-mori	n-tuples	13,422
mahiyama/quiz-works	n-tuples	12,502
mahiyama/amagasaki-qna	n-tuples	11,069
mahiyama/miracl-retrieval	n-tuples	4,603
mahiyama/mrtydi	n-tuples	3,602
mahiyama/anlp-meeting-retrieval (title-abs)	n-tuples	1,926
mahiyama/anlp-meeting-retrieval (title-intro)	n-tuples	1,952
mahiyama/civicqa-ja (non-public)	pairs	28,653
mahiyama/amagasaki-qna	pairs	7,323
mahiyama/kosodate-faq-pairs-ja	pairs	595
Total		379,029

mahiyama/civicqa-ja consists of questions and answers collected from FAQs published by Japanese government agencies and local governments. It is currently non-public.

Data used for evaluation was excluded from training. JaGovFaqs-22k was not used in training at all. Rows where the query after NFKC normalization and whitespace removal matched a query in the evaluation split of JaGovFaqs-22k, or where the positive after the same processing matched a document in that evaluation split, were excluded at each stage.

Training Procedure

The loss is computed for each of the six usable combinations and averaged. This is the Matryoshka training. The loss for each combination has the same structure as in the original model, adding a contrastive loss and a distillation loss.

• Contrastive loss — Cross-entropy with all documents in the batch (its own 6 candidates and documents from other rows in the same batch) as candidates.
• Distillation loss — KL divergence between the probability distribution of MeanMaxSim scores over its own 6 candidates and the probability distribution of teacher scores. The teacher temperature is 2.0 and the student temperature is 0.05.

For pairs, a contrastive loss is used with documents from other rows in the batch as negatives.

Learning rates were divided by parameter type to update the newly added summary tokens and projection layer faster, and the pre-trained Transformer slower.

Setting	Value
learning_rate (Transformer)	2e-5
learning_rate (summary tokens)	1e-4
learning_rate (projection layer)	1e-4
num_train_epochs	1
per_device_train_batch_size	64
lr_scheduler_type	linear
warmup_steps	5% of total
weight_decay	0.01
Input length during training	128 tokens + 16 for queries / 512 tokens + 64 for documents
Mixed precision	bf16
batch_sampler	NO_DUPLICATES
Gradient checkpointing	Enabled
Teacher temperature / student temperature	2.0 / 0.05
Random seed	42

The student temperature is set to 0.05 because MeanMaxSim divides by the number of vectors, resulting in a narrower range of scores than when using the sum.

Training took about 5 hours 6 minutes (5,923 steps) on a single NVIDIA RTX PRO 6000 Blackwell Server Edition (96 GB) GPU, with peak GPU memory usage of 23.5 GB.

5. Training Code with Sentence Transformers

The training workflow is the same as in Sazanami-ColBERT-310m; what differs is how the model is constructed and how the loss is computed. The differences are presented here.

MetaEmbedTransformer is a class in meta_modules.py, bundled in this repository. It inherits from sentence-transformers' Transformer and returns representations only at the positions of the summary tokens.

import torch
from sentence_transformers import MultiVectorEncoder
from sentence_transformers.base.modules import Dense, Normalize

from meta_modules import MetaEmbedTransformer

BASE = "sbintuitions/modernbert-ja-310m"
BASE_DIR = "base-mve"
QUERY_PREFIX = "検索クエリ: "
DOCUMENT_PREFIX = "検索文書: "

# Load the raw base model once into MultiVectorEncoder and save it.
# This puts the initialized projection layer (768 -> 128, no bias) into the folder.
MultiVectorEncoder(
    BASE,
    model_kwargs={"torch_dtype": torch.float32},
    similarity_fn_name="maxsim",
    device="cpu",
).save_pretrained(BASE_DIR)

# Transformer handling summary tokens. Add the number of summary tokens to the input length.
transformer = MetaEmbedTransformer(
    BASE_DIR,
    num_query_meta=16,
    num_document_meta=64,
    model_kwargs={"torch_dtype": torch.float32},
    query_length=256 + 16,
    document_length=1024 + 64,
)

# Sequences of summary tokens are prepended for both queries and documents.
prompts = {
    "query": transformer.query_prompt_prefix + QUERY_PREFIX,
    "document": transformer.document_prompt_prefix + DOCUMENT_PREFIX,
}

model = MultiVectorEncoder(
    modules=[
        transformer,
        Dense.load(BASE_DIR, subfolder="1_Dense"),
        Normalize(module_input_name="token_embeddings"),
    ],
    prompts=prompts,
    similarity_fn_name="meanmaxsim",   # Divide by the number of query vectors
)

From this point on, the training code from the Sazanami-ColBERT-310m card can be used as-is. Enabling gradient checkpointing, specifying column prefixes, calling MultiVectorEncoderTrainer, and saving the model are all the same. The following three points differ:

• For the loss, the contrastive loss and distillation loss are computed for each of the six usable combinations and averaged. The loss for each combination has the same structure as ContrastiveDistillLoss, using MeanMaxSim for scores and setting the student temperature to 0.05.
• Add an entry for summary tokens to learning_rate_mapping: {r"0\.(query_meta|document_meta)$": 1e-4, r"\.1\.linear\.": 1e-4}.
• Add the number of summary tokens to max_length: {"query": 128 + 16, "document": 512 + 64}.

6. Evaluation Results

Among the 11 Retrieval tasks included in JMTEB(v2) in MTEB 2.21.8, three tasks differing in document length and candidate count were evaluated. The metric is NDCG@10. The setting uses 1,024 tokens for documents and 256 tokens for queries. For this model, the summary tokens are added on top of these lengths.

Benchmark Size

Number of documents, queries, and document lengths for the three tasks.

Task	Documents	Queries	Avg. document tokens
Mintaka	1,592	2,312	10.0
JaGovFaqs	22,794	2,048	100.5
Jaqket	114,229	997	741.4

Average document tokens is the value calculated from the number of vectors actually generated by Sazanami-ColBERT-310m, which retains all tokens (1 vector = 1 token). It was measured with documents set to 1,024 tokens.

Retrieval Accuracy

Model	Mintaka	JaGovFaqs	Jaqket
Sazanami-ColBERT-310m	0.18070	0.77986	0.77868
Sazanami-AGC-310m	0.17302	0.78513	0.72986
Sazanami-MetaEmbed-310m (this model)	0.22697	0.79316	0.72855
Sazanami-Embed-310m	0.21771	0.78883	0.69682

The values for this model in the table above are for the combination of 16 query vectors and 64 document vectors. The table below shows NDCG@10 on JaGovFaqs when fewer vectors are used, keeping only the leading vectors of the encoded output:

Number of query vectors	Number of document vectors	NDCG@10
16	64	0.79316
8	32	0.77004
8	16	0.77227
4	8	0.76868
2	4	0.75971
1	1	0.75705

The relationship that accuracy increases as the number of vectors increases does not strictly hold. Even when increasing the document side from 16 to 32 vectors, there is no improvement on this data.

Storage Size and Similarity Computation Time

These are measured values on Jaqket with 114,229 documents. Documents were set to 1,024 tokens. Encoding is the process of converting document text into vectors, and is the total time to process all documents divided by 114,229 (as it is a batch process when building the index, it is averaged over GPU batch processing). Similarity computation time is the total time to process all 997 evaluation queries divided by 997.

Model	Vectors per document	Dimensions	Storage ratio	Encoding per document	Similarity computation per query
Sazanami-ColBERT-310m	741.4	128	1	11.57 ms	43.03 ms
Sazanami-AGC-310m	32.0	128	About 1/23	12.22 ms	4.01 ms
Sazanami-MetaEmbed-310m (this model)	64.0	128	About 1/12	12.56 ms	0.70 ms
Sazanami-Embed-310m	1.0	768	About 1/124	12.83 ms	0.90 ms

7. Discussion of the Results

• On tasks with short documents, it outperforms the model that retains all tokens. On JaGovFaqs, where documents average 100 tokens, this model's NDCG@10 is 0.79316, 0.013 higher than Sazanami-ColBERT-310m (0.77986). On Mintaka, where documents average 10 tokens, it is 0.046 higher at 0.22697, the highest among the four models presented. This can be interpreted as the approach of summarizing the entire document into a small number of vectors being advantageous for short documents.
• On tasks with long documents and many candidates, accuracy decreases. On Jaqket, with an average document length of 741 tokens and 114,229 documents, it is 0.72855, 0.050 lower than Sazanami-ColBERT-310m (0.77868).
• The difference from the single-vector model is small, being only 0.009 higher on Mintaka and 0.004 higher on JaGovFaqs. The gap widens on long documents, where it is 0.032 higher on Jaqket. For corpora consisting mainly of short documents, a single vector performs about as well, and the advantage of multi-vector models appears when searching for long documents among many candidates. However, on Jaqket, Sazanami-ColBERT-310m outperforms this model by an additional 0.050.
• Under the same condition of 32 document vectors, it falls short of Sazanami-AGC-310m. On JaGovFaqs, the configuration with 8 query vectors and 32 document vectors achieves 0.77004, which is 0.015 lower than Sazanami-AGC-310m (0.78513). This model outperforms it only when the maximum configuration can be used.

8. License and Training Data Restrictions

The model weights are released under the MIT License. The base model, sbintuitions/modernbert-ja-310m, is also released under the MIT License.

The training data includes a Japanese translation of MS MARCO (mahiyama/mmarco-ja). MS MARCO is released for non-commercial research purposes, and its license for commercial use is not clear. If you use this model commercially, please check the MS MARCO terms of use at your own responsibility.

Part of the training data includes questions and answers collected from FAQs published by Japanese government agencies and local governments. This data is subject to the terms of use of each source and is not redistributed in this model's repository. If you use this model, please determine the suitability for your application at your own responsibility.