---
language:
- ko
- en
license: apache-2.0
library_name: transformers
pipeline_tag: token-classification
base_model: openai/privacy-filter
tags:
- token-classification
- ner
- pii
- privacy
- pii-masking
- korean
- english
- finance
- customer-service
- security
- infrastructure
- bioes
- viterbi
- mixture-of-experts
datasets:
- BCCard/privacy-filter-openpii-masking
metrics:
- precision
- recall
- f1
---

# MoAI-Privacy-Filter
`MoAI-Privacy-Filter` is a Korean and English privacy-related entity detection model built by full fine-tuning [`openai/privacy-filter`](https://huggingface.co/openai/privacy-filter). It recognizes 29 entity types and emits 117 BIOES token classes. The training data emphasizes financial services and customer-service/VOC text while also covering identity, security, and infrastructure scenarios.

The model detects entity spans but does not decide how they should be masked or retained. Applications can apply their own handling policy to each predicted label. This distinction is especially important for `PORT` and `ORGANIZATION`, which are non-PII disambiguation labels included in the output taxonomy.

On held-out validation, strict micro F1 is **0.9824 for ko** and **0.9708 for en**. On an independently generated Golden Set, strict micro F1 is **0.9732 for ko** and **0.9650 for en**.

## 1. Model Summary

| Item | Value |
|---|---|
| Model version | v3 |
| Training dataset | [`BCCard/privacy-filter-openpii-masking`](https://huggingface.co/datasets/BCCard/privacy-filter-openpii-masking) v1 |
| Base model | [`openai/privacy-filter`](https://huggingface.co/openai/privacy-filter) |
| Architecture | Approximately 1.4B-parameter MoE, 8 layers, hidden size 640, 128 local experts, top-4 expert routing |
| Task | Token classification with BIOES span boundaries |
| Languages | Korean and English |
| Primary domains | Financial services, customer service/VOC, identity, security, and infrastructure |
| Entity labels | 29 |
| Output classes | 117 - `O` plus four BIOES classes for each entity label |
| Training sequence limit | 1024 tokens before special tokens |
| Artifact format | BF16 safetensors with 8 attention `sinks` tensors retained in FP32 |
| Tested software | Transformers 5.13.1 and PyTorch 2.13.0 |
| License | Apache 2.0 |

The model version and dataset version use independent version numbers. This model is v3 and was trained on dataset v1.

## 2. Label Taxonomy

| Label | Definition |
|---|---|
| `PERSON` | Full personal name as one span. |
| `RRN` | Korean resident registration number. |
| `FRN` | Korean foreign resident registration number. |
| `SSN` | Social-security-number family inherited from ai4privacy `SOCIALNUM`; not limited to the US 9-digit form. |
| `GENERIC_ID` | Identity-card or tax identifier that cannot be assigned to a more specific country-level label. |
| `CARD_NUMBER` | Credit or debit card PAN. |
| `ACCOUNT_NUMBER` | Bank account number. |
| `SECRET` | Password, API key, access token, or similar authentication secret. |
| `USER_ID` | Online account or member identifier. |
| `EMAIL` | Email address, including intentionally obfuscated forms represented in the training data. |
| `PHONE` | Mobile or landline telephone number, including intentionally verbalized forms represented in the training data. |
| `PASSPORT` | Passport number. |
| `DRIVER_LICENSE` | Driver's license number. |
| `ADDRESS` | City, street, and building components represented as one address span. |
| `ZIPCODE` | Postal code kept separate from `ADDRESS`. |
| `DATE` | Date or time. Timezone-only strings are not included in this label. |
| `CARD_EXPIRY` | Payment-card expiration date. |
| `CVC` | Card verification code. |
| `IPIN` | Korean I-PIN identifier. |
| `TRANSACTION_APPROVAL_ID` | Payment authorization or transaction approval identifier. |
| `BUSINESS_ID` | Business registration number or merchant identifier. |
| `VIRTUAL_CARD_NUMBER` | Alternate or virtual card number. |
| `CI` | Korean identity-linkage information value. |
| `IPADDRESS` | IPv4 network address. |
| `MACADDRESS` | 48-bit MAC address. |
| `IMEI` | Mobile-equipment identifier. |
| `PORT` | Network service port from 0 to 65535; a non-PII disambiguation label. |
| `ORGANIZATION` | Company, bank, hospital, or other organization name; a non-PII disambiguation label. |
| `URL` | Full web URL, including path, query, and fragment when present. |

Each entity label has `B-`, `I-`, `E-`, and `S-` boundary classes. Together with `O`, the model therefore has `4 x 29 + 1 = 117` output classes. `O` means that the model predicts no taxonomy entity at that token; it does not guarantee that the surrounding text is non-sensitive.

## 3. Usage

```python
import torch
from transformers import AutoModelForTokenClassification, AutoTokenizer

model_id = "BCCard/MoAI-Privacy-Filter"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForTokenClassification.from_pretrained(model_id)
model.eval()

text = "고객 모아이님(900101-1234569)께서 010-0000-0000로 연락 요청하셨습니다."
encoded = tokenizer(
    text,
    return_offsets_mapping=True,
    add_special_tokens=False,
    truncation=True,
    max_length=1024,
    return_tensors="pt",
)
offsets = encoded.pop("offset_mapping")[0].tolist()

with torch.no_grad():
    logits = model(**encoded).logits.float()

print(tuple(logits.shape))  # (1, sequence_length, 117)
```

Apply constrained BIOES Viterbi decoding to `logits[0]`, then map the decoded token spans to the original text with `offsets`. Character-span records can then be represented in the following form.

```text
[
  {'start': 3, 'end': 6, 'label': 'PERSON'},
  {'start': 8, 'end': 22, 'label': 'RRN'},
  {'start': 26, 'end': 39, 'label': 'PHONE'}
]
```

Character offsets use Python's half-open interval `[start, end)`. A downstream application could render those spans as follows, but this replacement behavior is not part of the model.

```text
고객 [PERSON]님([RRN])께서 [PHONE]로 연락 요청하셨습니다.
```

### 3.1. Decoding

The reported metrics use constrained Viterbi decoding over the BIOES transition grammar, followed by whitespace boundary refinement. Independent per-token argmax can emit invalid BIOES sequences and is not the reported decoding path.

The bundled `viterbi_calibration.json` contains six transition biases. The default operating point sets all biases to zero, so BIOES transition constraints remain active without an additional precision-recall adjustment. Convert logits to FP32 before decoding.

Use a decoder that implements this BIOES constraint contract. The upstream [`openai/privacy-filter`](https://github.com/openai/privacy-filter) project provides the reference implementation and decoding behavior on which this model is based.

For batches, use right padding and pass only `input_ids` and `attention_mask` to the model. Keep `offset_mapping` outside the model for character-span reconstruction. Inputs longer than 1024 tokens were not represented in the training regime and should be chunked with enough overlap for the target use case.

## 4. Training

### 4.1. Data

| Dataset | Role | Size |
|---|---|---:|
| [`BCCard/privacy-filter-openpii-masking`](https://huggingface.co/datasets/BCCard/privacy-filter-openpii-masking) v1 | Training | 57,851 rows |
| [`BCCard/privacy-filter-openpii-masking`](https://huggingface.co/datasets/BCCard/privacy-filter-openpii-masking) v1 | Validation and checkpoint selection | 14,524 rows |
| Independent non-public Golden Set | Final evaluation only | 4,000 rows - ko 2,920 and en 1,080 |

The Golden Set was not used for training, checkpoint selection, or calibration. The training and validation data contain all 29 labels in both languages. English represents 25.74% of train and 26.03% of validation.

The dataset combines relabeled rows from [`ai4privacy/pii-masking-openpii-1.5m`](https://huggingface.co/datasets/ai4privacy/pii-masking-openpii-1.5m) with Korean perturbation, English replay, and statically authored synthesis rows. Synthesis covers positive, confusion, hard-negative, weak-cue, long-context, and multi-label scenarios. The dataset is designed as synthetic training data and contains no operational customer records.

### 4.2. Procedure

<div align="center">
  <img src="figures/evaluation-train-1-1.png" alt="Training loss, learning rate, and gradient norm by training step">
</div>

| Item | Value |
|---|---|
| Method | Full fine-tuning of all parameters, including experts and router |
| Classification head | 33 base classes expanded to 117 classes through taxonomy-aware initialization |
| Loss | Token-level cross-entropy |
| Sequence length | Maximum 1024 tokens before special tokens |
| Batch | Per-device 16, effective 16 on one GPU |
| Learning rate | 1e-4 with linear decay |
| Warmup | 543 steps, approximately 3% of 18,080 total steps |
| Optimizer | Fused AdamW, weight decay 0.0, maximum gradient norm 1.0 |
| Epochs | 5, with the best checkpoint selected by validation strict micro F1 |
| Precision | FP32 master parameters with BF16 autocast; MoE router and expert compute protected in FP32 |
| Seed | 42 for training and data sampling |
| Hardware | 1 NVIDIA H100 80 GB GPU |

The selected FP32 checkpoint was exported as a deployment artifact in BF16. Artifact validation found 132 BF16 tensors and 8 FP32 attention `sinks` tensors.

### 4.3. Included Files

| File | Purpose |
|---|---|
| `model.safetensors` | BF16 deployment weights with FP32 attention `sinks`. |
| `config.json` | Architecture and 117-class label mapping. |
| `tokenizer.json` and `tokenizer_config.json` | Tokenizer files associated with the base model. |
| `label-taxonomy.yaml` | Label definitions and source-to-target mapping used for training. |
| `viterbi_calibration.json` | Constrained-decoding operating-point biases. |

## 5. Evaluation

### 5.1. Setup

The validation split was used for checkpoint selection and experiment comparison. Final generalization was measured on a separately generated Golden Set containing weak-context entities, surface-similar decoys, label-confusion pairs, and boundary variants.

The headline metrics are language-slice strict micro Precision, Recall, and F1. A predicted entity is correct only when both its label and complete span boundary match the reference. Golden evaluation uses constrained Viterbi decoding and whitespace boundary refinement.

### 5.2. Results

<div align="center">
  <img src="figures/evaluation-test-1-1.png" alt="Validation precision, recall, micro F1, and macro F1 by training step">
</div>

| Language | Validation P | Validation R | Validation F1 | Golden P | Golden R | Golden F1 | F1 Difference |
|---|---:|---:|---:|---:|---:|---:|---:|
| ko | 0.9829 | 0.9819 | 0.9824 | 0.9735 | 0.9729 | 0.9732 | -0.92%p |
| en | 0.9708 | 0.9708 | 0.9708 | 0.9652 | 0.9649 | 0.9650 | -0.58%p |

`F1 Difference` is Golden F1 minus Validation F1. The smaller Golden scores indicate a limited generalization decrease of 0.92 percentage points for Korean and 0.58 percentage points for English.

### 5.3. Error Characteristics

The aggregate results do not mean that every label performs equally. Error analysis of the English Golden slice shows the most visible weaknesses in `ACCOUNT_NUMBER`, `ZIPCODE`, `PORT`, and `ORGANIZATION`. Frequent confusion directions include `PORT` versus `ZIPCODE`, `ACCOUNT_NUMBER` versus `BUSINESS_ID` or `IPIN`, and `ORGANIZATION` versus `PERSON`.

These patterns are consistent with labels that share numeric shapes or require contextual role information. Downstream systems should evaluate per-label behavior on their own traffic, especially when label identity changes the handling action.

## 6. Intended Use

Suitable uses include:

* Detecting privacy-related entities before Korean or English text is sent to an LLM or another downstream service.
* Supporting offline privacy review of customer-service text, documents, email, and logs.
* Producing typed entity spans for an application-specific masking, routing, retention, or review policy.

The model is not a complete anonymization system, a legal-compliance guarantee, or a substitute for domain-specific review. It should not be used as the sole control for high-impact decisions. Applications remain responsible for deciding whether each detected label is masked, transformed, retained, or escalated.

## 7. Limitations

* **Synthetic evaluation** - Training, validation, and Golden data are synthetic. Performance on real customer text, OCR noise, slang, novel obfuscation, and unseen document structures has not been established.
* **Label-specific variation** - High aggregate F1 can hide weaker labels and confusion pairs. `ACCOUNT_NUMBER`, `ZIPCODE`, `PORT`, and `ORGANIZATION` require particular attention based on the current Golden analysis.
* **Non-PII labels** - `PORT` and `ORGANIZATION` are deliberately predicted even though they are not PII. Consumers must not assume that every non-`O` label requires the same action.
* **Context and boundary sensitivity** - Weak contextual evidence, shared numeric formats, and long entity boundaries can produce missed entities, boundary errors, or label swaps.
* **Long inputs** - The base architecture supports a larger context, but training examples were limited to 1024 tokens and the observed dataset maximum was 801 tokens. Longer inputs require separate validation and should normally be chunked.
* **Registry-backed identifiers** - Synthetic account, telephone, passport, user, and social-security values cannot be exhaustively checked against private issuance registries. Any coincidental match with a real value is unintended.
* **Language and domain scope** - Evaluation covers Korean and English with emphasis on financial, customer-service/VOC, identity, security, and infrastructure contexts. Other languages and domains are unsupported.

## 8. License, Attribution, and Citation

The model is released under the Apache 2.0 license. Its training dataset is released under CC BY 4.0 and is derived from `ai4privacy/pii-masking-openpii-1.5m`; follow the dataset card for its attribution requirements.

```bibtex
@misc{bccard2026moaiprivacyfilterv3,
  title        = {MoAI-Privacy-Filter v3: Korean and English Privacy-Related Entity Detection},
  author       = {BC Card},
  year         = {2026},
  howpublished = {https://huggingface.co/BCCard/MoAI-Privacy-Filter},
  note         = {Full fine-tune of openai/privacy-filter on BCCard/privacy-filter-openpii-masking v1}
}
```

Related resources:

* **Base model**: [`openai/privacy-filter`](https://huggingface.co/openai/privacy-filter).
* **Training dataset**: [`BCCard/privacy-filter-openpii-masking`](https://huggingface.co/datasets/BCCard/privacy-filter-openpii-masking).
* **Upstream dataset**: [`ai4privacy/pii-masking-openpii-1.5m`](https://huggingface.co/datasets/ai4privacy/pii-masking-openpii-1.5m).

## 9. Disclaimer

This model is provided as is, without warranties of accuracy, completeness, non-infringement, or fitness for a particular purpose. Users are responsible for testing the model in their own environment and ensuring that its use complies with applicable laws, regulations, contractual obligations, and organizational policies.