Upload folder using huggingface_hub

README.md

@@ -0,0 +1,137 @@
+---
+language:
+- rna
+library_name: transformers
+tags:
+- RNA
+- language-model
+- bert
+license: mit
+---
+
+# RNABERT
+
+A small BERT-style RNA language model pretrained on non-coding RNA sequences from Rfam 14.3, using
+Masked Language Modeling (MLM) and Structural Alignment Learning (SAL). Designed for RNA clustering
+and structural alignment tasks.
+
+## Architecture
+
+| Parameter | Value |
+|---|---|
+| Layers | 6 |
+| Attention heads | 12 |
+| Embedding dimension | 120 |
+| FFN intermediate size | 40 |
+| Vocabulary size | 6 (PAD, MASK, A, U, G, C) |
+| Positional encoding | Learned absolute |
+| Architecture | Post-LN BERT encoder |
+| Max sequence length | 440 |
+
+**Vocabulary:**
+
+| Token | ID |
+|---|---|
+| `<pad>` | 0 |
+| `<mask>` | 1 |
+| A | 2 |
+| U | 3 |
+| G | 4 |
+| C | 5 |
+
+No CLS or EOS tokens are added. Sequences are tokenized character-by-character; T is silently converted to U.
+
+## Pretraining
+
+- **Objective:** Masked Language Modeling (MLM) + Structural Alignment Learning (SAL, a pairwise
+  structural alignment contrastive objective)
+- **Data:** Rfam 14.3 (~440 nt max length sequences)
+- **Source checkpoint:** `bert_mul_2.pth` (distributed inside `RNABERT_pretrained.pth` zip)
+
+### Checkpoint selection
+
+There is one published pretrained checkpoint from the original repository. This is it.
+
+## Parity Verification
+
+Hidden-state representations verified identical (max abs diff = 2.2e-6) to the original
+implementation at all 7 representation levels (embedding + 6 transformer layers), with and
+without padding. Verified on CPU with PyTorch 2.7 / transformers 4.57.6.
+
+## Related Models
+
+See the full [RNABERT collection](https://huggingface.co/collections/Taykhoom/rnabert-PLACEHOLDER).
+
+| Model | Notes |
+|---|---|
+| **[Taykhoom/RNABERT](https://huggingface.co/Taykhoom/RNABERT)** | This model |
+
+## Usage
+
+### Embedding generation
+
+```python
+import torch
+from transformers import AutoTokenizer, AutoModel
+
+tokenizer = AutoTokenizer.from_pretrained("Taykhoom/RNABERT", trust_remote_code=True)
+model = AutoModel.from_pretrained("Taykhoom/RNABERT")
+model.eval()
+
+sequences = ["AUGCAUGCAUGC", "GCUAGCUAGCUA"]
+enc = tokenizer(sequences, return_tensors="pt", padding=True)
+
+with torch.no_grad():
+    out = model(**enc)
+
+# Token-level embeddings
+token_emb = out.last_hidden_state   # (batch, seq_len, 120)
+
+# Mean-pool over non-padding positions
+mask = enc["attention_mask"].unsqueeze(-1).float()
+mean_emb = (token_emb * mask).sum(1) / mask.sum(1)  # (batch, 120)
+
+# Intermediate layers
+out_all = model(**enc, output_hidden_states=True)
+layer3_emb = out_all.hidden_states[3]   # (batch, seq_len, 120)
+```
+
+### Fine-tuning
+
+Standard HF conventions. The model has no CLS token, so use mean pooling over non-padding
+positions for sequence-level tasks.
+
+## Implementation Notes
+
+This model uses the standard HuggingFace `BertModel` (`model_type: "bert"`) with custom
+hyperparameters matching the original RNABERT architecture. No custom modeling code is required;
+`trust_remote_code=True` is only needed for the tokenizer.
+
+The original implementation uses standard scaled dot-product attention (post-LN BERT). This HF
+port adds `attn_implementation="sdpa"` and `attn_implementation="flash_attention_2"` support via
+the standard HF dispatch mechanism, which were not part of the original codebase.
+
+## Citation
+
+```bibtex
+@article{akiyama2022informative,
+  title={Informative {RNA}-base embedding for functional {RNA} clustering and structural alignment},
+  author={Akiyama, Manato and Hamada, Michiaki},
+  journal={NAR Genomics and Bioinformatics},
+  volume={4},
+  number={1},
+  pages={lqac012},
+  year={2022},
+  publisher={Oxford University Press}
+}
+```
+
+## Credits
+
+Original model and code by Akiyama and Hamada. Source: [GitHub](https://github.com/mana438/RNABERT).
+The HF conversion code was authored primarily by [Claude Code](https://claude.ai/code)
+and reviewed manually by Taykhoom Dalal.
+
+## License
+
+MIT, following the original repository.

config.json

@@ -0,0 +1,24 @@
+{
+  "architectures": [
+    "BertModel"
+  ],
+  "attention_probs_dropout_prob": 0.0,
+  "classifier_dropout": null,
+  "dtype": "float32",
+  "hidden_act": "gelu",
+  "hidden_dropout_prob": 0.0,
+  "hidden_size": 120,
+  "initializer_range": 0.02,
+  "intermediate_size": 40,
+  "layer_norm_eps": 1e-12,
+  "max_position_embeddings": 440,
+  "model_type": "bert",
+  "num_attention_heads": 12,
+  "num_hidden_layers": 6,
+  "pad_token_id": 0,
+  "position_embedding_type": "absolute",
+  "transformers_version": "4.57.6",
+  "type_vocab_size": 2,
+  "use_cache": true,
+  "vocab_size": 6
+}

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:3f2e501e6923a5d0aeb67ea4709d2a796b583e81057602b719689a8a533743da
+size 1924760

special_tokens_map.json

@@ -0,0 +1,5 @@
+{
+  "pad_token": "<pad>",
+  "mask_token": "<mask>",
+  "unk_token": "<pad>"
+}

tokenization_rnabert.py

@@ -0,0 +1,69 @@
+import json
+import os
+from transformers import PreTrainedTokenizer
+
+VOCAB = {"<pad>": 0, "<mask>": 1, "A": 2, "U": 3, "G": 4, "C": 5}
+
+
+class RNABertTokenizer(PreTrainedTokenizer):
+    vocab_files_names = {"vocab_file": "vocab.json"}
+    model_input_names = ["input_ids", "attention_mask"]
+
+    def __init__(
+        self,
+        vocab_file=None,
+        pad_token="<pad>",
+        mask_token="<mask>",
+        unk_token="<pad>",
+        **kwargs,
+    ):
+        self._vocab = dict(VOCAB)
+        if vocab_file and os.path.isfile(vocab_file):
+            with open(vocab_file) as f:
+                self._vocab = json.load(f)
+        self._ids_to_tokens = {v: k for k, v in self._vocab.items()}
+        super().__init__(
+            pad_token=pad_token,
+            mask_token=mask_token,
+            unk_token=unk_token,
+            **kwargs,
+        )
+
+    @property
+    def vocab_size(self):
+        return len(self._vocab)
+
+    def get_vocab(self):
+        return dict(self._vocab)
+
+    def _tokenize(self, text):
+        return list(text.upper().replace("T", "U"))
+
+    def _convert_token_to_id(self, token):
+        return self._vocab.get(token, 0)
+
+    def _convert_id_to_token(self, index):
+        return self._ids_to_tokens.get(index, "<pad>")
+
+    def save_vocabulary(self, save_directory, filename_prefix=None):
+        os.makedirs(save_directory, exist_ok=True)
+        fname = (filename_prefix + "-" if filename_prefix else "") + "vocab.json"
+        path = os.path.join(save_directory, fname)
+        with open(path, "w") as f:
+            json.dump(self._vocab, f, indent=2)
+        return (path,)
+
+    def build_inputs_with_special_tokens(self, token_ids_0, token_ids_1=None):
+        if token_ids_1 is None:
+            return token_ids_0
+        return token_ids_0 + token_ids_1
+
+    def get_special_tokens_mask(self, token_ids_0, token_ids_1=None, already_has_special_tokens=False):
+        if already_has_special_tokens:
+            return super().get_special_tokens_mask(token_ids_0, token_ids_1, True)
+        return [0] * len(token_ids_0) + ([0] * len(token_ids_1) if token_ids_1 else [])
+
+    def create_token_type_ids_from_sequences(self, token_ids_0, token_ids_1=None):
+        if token_ids_1 is None:
+            return [0] * len(token_ids_0)
+        return [0] * len(token_ids_0) + [0] * len(token_ids_1)

tokenizer_config.json

@@ -0,0 +1,11 @@
+{
+  "auto_map": {
+    "AutoTokenizer": ["tokenization_rnabert.RNABertTokenizer", null]
+  },
+  "model_max_length": 440,
+  "tokenizer_class": "RNABertTokenizer",
+  "pad_token": "<pad>",
+  "mask_token": "<mask>",
+  "unk_token": "<pad>",
+  "vocab_file": "vocab.json"
+}

vocab.json

@@ -0,0 +1,8 @@
+{
+  "<pad>": 0,
+  "<mask>": 1,
+  "A": 2,
+  "U": 3,
+  "G": 4,
+  "C": 5
+}