Oaimgeo alex-medvedev-msc commited on
Commit
8b31bcc
·
0 Parent(s):

Duplicate from m42-health/BioFM-265M

Browse files

Co-authored-by: Aleksandr Medvedev <alex-medvedev-msc@users.noreply.huggingface.co>

.gitattributes ADDED
@@ -0,0 +1,40 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ *.7z filter=lfs diff=lfs merge=lfs -text
2
+ *.arrow filter=lfs diff=lfs merge=lfs -text
3
+ *.bin filter=lfs diff=lfs merge=lfs -text
4
+ *.bz2 filter=lfs diff=lfs merge=lfs -text
5
+ *.ckpt filter=lfs diff=lfs merge=lfs -text
6
+ *.ftz filter=lfs diff=lfs merge=lfs -text
7
+ *.gz filter=lfs diff=lfs merge=lfs -text
8
+ *.h5 filter=lfs diff=lfs merge=lfs -text
9
+ *.joblib filter=lfs diff=lfs merge=lfs -text
10
+ *.lfs.* filter=lfs diff=lfs merge=lfs -text
11
+ *.mlmodel filter=lfs diff=lfs merge=lfs -text
12
+ *.model filter=lfs diff=lfs merge=lfs -text
13
+ *.msgpack filter=lfs diff=lfs merge=lfs -text
14
+ *.npy filter=lfs diff=lfs merge=lfs -text
15
+ *.npz filter=lfs diff=lfs merge=lfs -text
16
+ *.onnx filter=lfs diff=lfs merge=lfs -text
17
+ *.ot filter=lfs diff=lfs merge=lfs -text
18
+ *.parquet filter=lfs diff=lfs merge=lfs -text
19
+ *.pb filter=lfs diff=lfs merge=lfs -text
20
+ *.pickle filter=lfs diff=lfs merge=lfs -text
21
+ *.pkl filter=lfs diff=lfs merge=lfs -text
22
+ *.pt filter=lfs diff=lfs merge=lfs -text
23
+ *.pth filter=lfs diff=lfs merge=lfs -text
24
+ *.rar filter=lfs diff=lfs merge=lfs -text
25
+ *.safetensors filter=lfs diff=lfs merge=lfs -text
26
+ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
27
+ *.tar.* filter=lfs diff=lfs merge=lfs -text
28
+ *.tar filter=lfs diff=lfs merge=lfs -text
29
+ *.tflite filter=lfs diff=lfs merge=lfs -text
30
+ *.tgz filter=lfs diff=lfs merge=lfs -text
31
+ *.wasm filter=lfs diff=lfs merge=lfs -text
32
+ *.xz filter=lfs diff=lfs merge=lfs -text
33
+ *.zip filter=lfs diff=lfs merge=lfs -text
34
+ *.zst filter=lfs diff=lfs merge=lfs -text
35
+ *tfevents* filter=lfs diff=lfs merge=lfs -text
36
+ biotoken_biofm.png filter=lfs diff=lfs merge=lfs -text
37
+ figures/expression_model_comparison.png filter=lfs diff=lfs merge=lfs -text
38
+ figures/nt_lr_heatmap_and_barh.png filter=lfs diff=lfs merge=lfs -text
39
+ figures/sqtl_model_comparison.png filter=lfs diff=lfs merge=lfs -text
40
+ figures/vb_heatmap_and_barh_max.png filter=lfs diff=lfs merge=lfs -text
README.md ADDED
@@ -0,0 +1,164 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ license: cc-by-nc-4.0
3
+ tags:
4
+ - m42
5
+ - genomics
6
+ - biology
7
+ - GFM
8
+ - BioFM
9
+ - BioToken
10
+ ---
11
+
12
+ # BioFM: A Biologically-Informed Genomic Foundation Model
13
+ BioFM is a cutting-edge genomic foundation model that addresses critical limitations in existing genomic sequence modeling. By introducing BioToken, a novel tokenization framework, BioFM encodes genomic variants and structural annotations with unprecedented biological context, enabling more nuanced and accurate representation learning.
14
+
15
+ ![BioFM](figures/biotoken_biofm.png)
16
+
17
+ ## Model Highlights
18
+ - With the introduction of BioToken, we achieved competitive genomic prediction results using only 265 million parameters, significantly reducing computational requirements and training costs.
19
+ - Demonstrated superior performance compared to specialized models like Enformer and SpliceTransformer in critical genomic tasks, such as expression prediction and sQTL prediction, respectively.
20
+ - BioFM excels at various genomic tasks (e.g., expression prediction, coding/non-coding pathogenicity prediction, and sQTL prediction) that require long-range genomic contexts, outperforming existing GFMs.
21
+
22
+ ## Model Details
23
+ - **Model developers:** M42 Health AI Team
24
+ - **Base architecture:** [MistralForCausalLM](https://huggingface.co/docs/transformers/main/en/model_doc/mistral#transformers.MistralForCausalLM)
25
+ - **Context length:**
26
+ - **Training:** 6k tokens
27
+ - **Inference:** 12k tokens
28
+ - **Training data:** 1000 Genomes
29
+ - **Input format:** Annotated DNA sequences using BioToken
30
+ - **Output options:**
31
+ - DNA sequences only
32
+ - Embeddings
33
+ - **License:** CC BY-NC 4.0
34
+ - **Publication:** [Paper link]()
35
+
36
+ ## Model Inference
37
+ We developed a BioFM-Eval Python package for inference and embedding extraction from genomic sequences. Refer to [BioFM-Eval](https://github.com/m42-health/biofm-eval/) library for setup and installation instructions.
38
+
39
+ ### Creating Variant Embeddings with BioFM
40
+
41
+ This guide will help you quickly generate BioFM embeddings for the variants in your VCF file. These embeddings are created using the method described in our publication.
42
+
43
+ ```python
44
+ from biofm_eval import AnnotatedModel, AnnotationTokenizer, Embedder, VCFConverter
45
+ import torch
46
+
47
+ # Define paths to the pre-trained BioFM model and tokenizer
48
+ MODEL_PATH = "m42-health/BioFM-265M"
49
+ TOKENIZER_PATH = "m42-health/BioFM-265M"
50
+
51
+ # Load the pre-trained BioFM model and BioToken tokenizer
52
+ model = AnnotatedModel.from_pretrained(
53
+ MODEL_PATH,
54
+ torch_dtype=torch.bfloat16,
55
+ )
56
+ tokenizer = AnnotationTokenizer.from_pretrained(TOKENIZER_PATH)
57
+
58
+ # Initialize the embedder using the model and tokenizer
59
+ embedder = Embedder(model, tokenizer)
60
+
61
+ # Set up the VCF converter with paths to gene annotations and reference genome
62
+ vcf_converter = VCFConverter(
63
+ gene_annotation_path="./gencode.v38.annotation.gff3",
64
+ reference_genome_path="./GCA_000001405.15_GRCh38_no_alt_plus_hs38d1_analysis_set.fna"
65
+ )
66
+
67
+ # Convert a VCF file into an annotated dataset using BioTokens
68
+ annotated_dataset = vcf_converter.vcf_to_annotated_dataset(
69
+ vcf_path = './HG01779_b.vcf.gz',
70
+ max_variants=200 # Set to None to process all variants in the VCF file
71
+ )
72
+
73
+ # Extract BioFM embeddings for all annotated variants
74
+ embeddings = embedder.get_dataset_embeddings(annotated_dataset)
75
+ print(embeddings)
76
+
77
+ # Example output (dict):
78
+ # {
79
+ # 'embeddings': array of shape (num_variants, 2*embedding_dim), # Numeric embeddings for each variant
80
+ # 'labels': array of shape (num_variants,) # Present only during supervised embedding extraction
81
+ # }
82
+
83
+ ```
84
+ - Sample reference genome fasta file: [download link](https://www.ncbi.nlm.nih.gov/datasets/genome/GCF_000001405.26/)
85
+ - Gene annotation file: [download_link](https://www.gencodegenes.org/human/release_38.html)
86
+ - Sample vcf file from 1000 Genomes data: [download_link](https://ftp.1000genomes.ebi.ac.uk/vol1/ftp/release/20130502/)
87
+
88
+
89
+ ### Generation with BioFM
90
+ BioFM can generate genomic sequences based on input DNA prompts.
91
+
92
+ ```python
93
+ from biofm_eval import AnnotatedModel, AnnotationTokenizer, Generator
94
+ import torch
95
+
96
+ # Define paths to the pre-trained BioFM model and tokenizer
97
+ MODEL_PATH = "m42-health/BioFM-265M"
98
+ TOKENIZER_PATH = "m42-health/BioFM-265M"
99
+
100
+ # Load the pre-trained BioFM model and BioToken tokenizer
101
+ model = AnnotatedModel.from_pretrained(
102
+ MODEL_PATH,
103
+ torch_dtype=torch.bfloat16,
104
+ )
105
+ tokenizer = AnnotationTokenizer.from_pretrained(TOKENIZER_PATH)
106
+
107
+ # Initializing the generator using model and tokenizer
108
+ seq_generator = Generator(model, tokenizer)
109
+
110
+ # Generate DNA sequences
111
+ input_sequences = ['AGCT', 'GACTGCA']
112
+ output = seq_generator.generate(
113
+ input_sequences,
114
+ max_new_tokens=10,
115
+ temperature=1.0,
116
+ do_sample=True,
117
+ top_k=4)
118
+
119
+ print(output)
120
+
121
+ # Example output: List[str] = ['AGCTACTCCCCTCC', 'GACTGCACCACTGTACT']
122
+
123
+ ```
124
+
125
+ ## Training Setup
126
+
127
+ The training was conducted on the NVIDIA DGX cluster with H100 GPUs, utilizing PyTorch's Fully Sharded Data Parallel (FSDP) framework.
128
+
129
+ ## Evaluation Results
130
+
131
+ To demonstrate the effectiveness of BioToken, we evaluated BioFM against strong supervised baselines: Enformer for gene expression prediction and Splice Transformer for sQTL prediction.
132
+
133
+ - *Gene Expression Prediction:* BioFM matches Enformer's performance when both models use a 12K context, making it the first-ever GFM to achieve this. Notably, Enformer fails to reach this performance level even with a 98K context.
134
+ - *sQTL Prediction:* BioFM significantly outperforms Splice Transformer across all tissues, highlighting its robustness and generalizability.
135
+
136
+ | sQTL prediction | Expression prediction |
137
+ |---------|---------|
138
+ | ![Alt1](figures/sqtl_model_comparison.png) | ![Alt2](figures/expression_model_comparison.png) |
139
+
140
+ We further evaluated BioFM on the Variant Benchmark we curated and the Genomics Long-Range Benchmark.
141
+
142
+ - *Variant Benchmark:* Across a broad spectrum of [variant prediction tasks](https://huggingface.co/datasets/m42-health/variant-benchmark), BioFM outperforms other GFMs, showcasing its superior predictive capabilities.
143
+ - *Long-Range Genomic Dependencies:* On the Genomics Long-Range Benchmark, BioFM sets new performance standards, surpassing previous GFMs that required extensive fine-tuning and longer genomic contexts. This highlights BioFM’s ability to effectively capture and utilize long-range genomic dependencies.
144
+
145
+ | Variant benchmark | Genomics long-range benchmark |
146
+ |---------|---------|
147
+ | ![Alt1](figures/vb_heatmap_and_barh_max.png) | ![Alt2](figures/nt_lr_heatmap_and_barh.png) |
148
+
149
+ Please go through the [paper]() for more resutls and ablations.
150
+
151
+ ## Citation
152
+ ```
153
+ @article {Medvedev2025.03.27.645711,
154
+ author = {Medvedev, Aleksandr and Viswanathan, Karthik and Kanithi, Praveenkumar and Vishniakov, Kirill and Munjal, Prateek and Christophe, Clement and Pimentel, Marco AF and Rajan, Ronnie and Khan, Shadab},
155
+ title = {BioToken and BioFM - Biologically-Informed Tokenization Enables Accurate and Efficient Genomic Foundation Models},
156
+ elocation-id = {2025.03.27.645711},
157
+ year = {2025},
158
+ doi = {10.1101/2025.03.27.645711},
159
+ publisher = {Cold Spring Harbor Laboratory},
160
+ URL = {https://www.biorxiv.org/content/early/2025/04/01/2025.03.27.645711},
161
+ eprint = {https://www.biorxiv.org/content/early/2025/04/01/2025.03.27.645711.full.pdf},
162
+ journal = {bioRxiv}
163
+ }
164
+ ```
config.json ADDED
@@ -0,0 +1,553 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "architectures": [
3
+ "MistralForCausalLM"
4
+ ],
5
+ "attention_dropout": 0.0,
6
+ "bos_token_id": 11,
7
+ "class_weights": [
8
+ 1.0,
9
+ 1.0,
10
+ 1.0,
11
+ 1.0,
12
+ 1.0,
13
+ 1.0,
14
+ 1.0,
15
+ 1.0,
16
+ 1.0,
17
+ 1.0,
18
+ 1.0,
19
+ 1.0,
20
+ 1.0,
21
+ 1.0,
22
+ 1.0,
23
+ 1.0,
24
+ 1.0,
25
+ 1.0,
26
+ 1.0,
27
+ 1.0,
28
+ 1.0,
29
+ 1.0,
30
+ 1.0,
31
+ 1.0,
32
+ 1.0,
33
+ 1.0,
34
+ 1.0,
35
+ 1.0,
36
+ 0.0,
37
+ 0.0,
38
+ 0.0,
39
+ 0.0,
40
+ 0.0,
41
+ 0.0,
42
+ 0.0,
43
+ 0.0,
44
+ 0.0,
45
+ 0.0,
46
+ 0.0,
47
+ 0.0,
48
+ 0.0,
49
+ 0.0,
50
+ 0.0,
51
+ 0.0,
52
+ 0.0,
53
+ 0.0,
54
+ 0.0,
55
+ 0.0,
56
+ 0.0,
57
+ 0.0,
58
+ 0.0,
59
+ 0.0,
60
+ 0.0,
61
+ 0.0,
62
+ 0.0,
63
+ 0.0,
64
+ 0.0,
65
+ 0.0,
66
+ 0.0,
67
+ 0.0,
68
+ 0.0,
69
+ 0.0,
70
+ 0.0,
71
+ 0.0,
72
+ 0.0,
73
+ 0.0,
74
+ 0.0,
75
+ 0.0,
76
+ 0.0,
77
+ 0.0,
78
+ 0.0,
79
+ 0.0,
80
+ 0.0,
81
+ 0.0,
82
+ 0.0,
83
+ 0.0,
84
+ 0.0,
85
+ 0.0,
86
+ 0.0,
87
+ 0.0,
88
+ 0.0,
89
+ 0.0,
90
+ 0.0,
91
+ 0.0,
92
+ 0.0,
93
+ 0.0,
94
+ 0.0,
95
+ 0.0,
96
+ 0.0,
97
+ 0.0,
98
+ 0.0,
99
+ 0.0,
100
+ 0.0,
101
+ 0.0,
102
+ 0.0,
103
+ 0.0,
104
+ 0.0,
105
+ 0.0,
106
+ 0.0,
107
+ 0.0,
108
+ 0.0,
109
+ 0.0,
110
+ 0.0,
111
+ 0.0,
112
+ 0.0,
113
+ 0.0,
114
+ 0.0,
115
+ 0.0,
116
+ 0.0,
117
+ 0.0,
118
+ 0.0,
119
+ 0.0,
120
+ 0.0,
121
+ 0.0,
122
+ 0.0,
123
+ 0.0,
124
+ 0.0,
125
+ 0.0,
126
+ 0.0,
127
+ 0.0,
128
+ 0.0,
129
+ 0.0,
130
+ 0.0,
131
+ 0.0,
132
+ 0.0,
133
+ 0.0,
134
+ 0.0,
135
+ 0.0,
136
+ 0.0,
137
+ 0.0,
138
+ 0.0,
139
+ 0.0,
140
+ 0.0,
141
+ 0.0,
142
+ 0.0,
143
+ 0.0,
144
+ 0.0,
145
+ 0.0,
146
+ 0.0,
147
+ 0.0,
148
+ 0.0,
149
+ 0.0,
150
+ 0.0,
151
+ 0.0,
152
+ 0.0,
153
+ 0.0,
154
+ 0.0,
155
+ 0.0,
156
+ 0.0,
157
+ 0.0,
158
+ 0.0,
159
+ 0.0,
160
+ 0.0,
161
+ 0.0,
162
+ 0.0,
163
+ 0.0,
164
+ 0.0,
165
+ 0.0,
166
+ 0.0,
167
+ 0.0,
168
+ 0.0,
169
+ 0.0,
170
+ 0.0,
171
+ 0.0,
172
+ 0.0,
173
+ 0.0,
174
+ 0.0,
175
+ 0.0,
176
+ 0.0,
177
+ 0.0,
178
+ 0.0,
179
+ 0.0,
180
+ 0.0,
181
+ 0.0,
182
+ 0.0,
183
+ 0.0,
184
+ 0.0,
185
+ 0.0,
186
+ 0.0,
187
+ 0.0,
188
+ 0.0,
189
+ 0.0,
190
+ 0.0,
191
+ 0.0,
192
+ 0.0,
193
+ 0.0,
194
+ 0.0,
195
+ 0.0,
196
+ 0.0,
197
+ 0.0,
198
+ 0.0,
199
+ 0.0,
200
+ 0.0,
201
+ 0.0,
202
+ 0.0,
203
+ 0.0,
204
+ 0.0,
205
+ 0.0,
206
+ 0.0,
207
+ 0.0,
208
+ 0.0,
209
+ 0.0,
210
+ 0.0,
211
+ 0.0,
212
+ 0.0,
213
+ 0.0,
214
+ 0.0,
215
+ 0.0,
216
+ 0.0,
217
+ 0.0,
218
+ 0.0,
219
+ 0.0,
220
+ 0.0,
221
+ 0.0,
222
+ 0.0,
223
+ 0.0,
224
+ 0.0,
225
+ 0.0,
226
+ 0.0,
227
+ 0.0,
228
+ 0.0,
229
+ 0.0,
230
+ 0.0,
231
+ 0.0,
232
+ 0.0,
233
+ 0.0,
234
+ 0.0,
235
+ 0.0,
236
+ 0.0,
237
+ 0.0,
238
+ 0.0,
239
+ 0.0,
240
+ 0.0,
241
+ 0.0,
242
+ 0.0,
243
+ 0.0,
244
+ 0.0,
245
+ 0.0,
246
+ 0.0,
247
+ 0.0,
248
+ 0.0,
249
+ 0.0,
250
+ 0.0,
251
+ 0.0,
252
+ 0.0,
253
+ 0.0,
254
+ 0.0,
255
+ 0.0,
256
+ 0.0,
257
+ 0.0,
258
+ 0.0,
259
+ 0.0,
260
+ 0.0,
261
+ 0.0,
262
+ 0.0,
263
+ 0.0,
264
+ 0.0,
265
+ 0.0,
266
+ 0.0,
267
+ 0.0,
268
+ 0.0,
269
+ 0.0,
270
+ 0.0,
271
+ 0.0,
272
+ 0.0,
273
+ 0.0,
274
+ 0.0,
275
+ 0.0,
276
+ 0.0,
277
+ 0.0,
278
+ 0.0,
279
+ 0.0,
280
+ 0.0,
281
+ 0.0,
282
+ 0.0,
283
+ 0.0,
284
+ 0.0,
285
+ 0.0,
286
+ 0.0,
287
+ 0.0,
288
+ 0.0,
289
+ 0.0,
290
+ 0.0,
291
+ 0.0,
292
+ 0.0,
293
+ 0.0,
294
+ 0.0,
295
+ 0.0,
296
+ 0.0,
297
+ 0.0,
298
+ 0.0,
299
+ 0.0,
300
+ 0.0,
301
+ 0.0,
302
+ 0.0,
303
+ 0.0,
304
+ 0.0,
305
+ 0.0,
306
+ 0.0,
307
+ 0.0,
308
+ 0.0,
309
+ 0.0,
310
+ 0.0,
311
+ 0.0,
312
+ 0.0,
313
+ 0.0,
314
+ 0.0,
315
+ 0.0,
316
+ 0.0,
317
+ 0.0,
318
+ 0.0,
319
+ 0.0,
320
+ 0.0,
321
+ 0.0,
322
+ 0.0,
323
+ 0.0,
324
+ 0.0,
325
+ 0.0,
326
+ 0.0,
327
+ 0.0,
328
+ 0.0,
329
+ 0.0,
330
+ 0.0,
331
+ 0.0,
332
+ 0.0,
333
+ 0.0,
334
+ 0.0,
335
+ 0.0,
336
+ 0.0,
337
+ 0.0,
338
+ 0.0,
339
+ 0.0,
340
+ 0.0,
341
+ 0.0,
342
+ 0.0,
343
+ 0.0,
344
+ 0.0,
345
+ 0.0,
346
+ 0.0,
347
+ 0.0,
348
+ 0.0,
349
+ 0.0,
350
+ 0.0,
351
+ 0.0,
352
+ 0.0,
353
+ 0.0,
354
+ 0.0,
355
+ 0.0,
356
+ 0.0,
357
+ 0.0,
358
+ 0.0,
359
+ 0.0,
360
+ 0.0,
361
+ 0.0,
362
+ 0.0,
363
+ 0.0,
364
+ 0.0,
365
+ 0.0,
366
+ 0.0,
367
+ 0.0,
368
+ 0.0,
369
+ 0.0,
370
+ 0.0,
371
+ 0.0,
372
+ 0.0,
373
+ 0.0,
374
+ 0.0,
375
+ 0.0,
376
+ 0.0,
377
+ 0.0,
378
+ 0.0,
379
+ 0.0,
380
+ 0.0,
381
+ 0.0,
382
+ 0.0,
383
+ 0.0,
384
+ 0.0,
385
+ 0.0,
386
+ 0.0,
387
+ 0.0,
388
+ 0.0,
389
+ 0.0,
390
+ 0.0,
391
+ 0.0,
392
+ 0.0,
393
+ 0.0,
394
+ 0.0,
395
+ 0.0,
396
+ 0.0,
397
+ 0.0,
398
+ 0.0,
399
+ 0.0,
400
+ 0.0,
401
+ 0.0,
402
+ 0.0,
403
+ 0.0,
404
+ 0.0,
405
+ 0.0,
406
+ 0.0,
407
+ 0.0,
408
+ 0.0,
409
+ 0.0,
410
+ 0.0,
411
+ 0.0,
412
+ 0.0,
413
+ 0.0,
414
+ 0.0,
415
+ 0.0,
416
+ 0.0,
417
+ 0.0,
418
+ 0.0,
419
+ 0.0,
420
+ 0.0,
421
+ 0.0,
422
+ 0.0,
423
+ 0.0,
424
+ 0.0,
425
+ 0.0,
426
+ 0.0,
427
+ 0.0,
428
+ 0.0,
429
+ 0.0,
430
+ 0.0,
431
+ 0.0,
432
+ 0.0,
433
+ 0.0,
434
+ 0.0,
435
+ 0.0,
436
+ 0.0,
437
+ 0.0,
438
+ 0.0,
439
+ 0.0,
440
+ 0.0,
441
+ 0.0,
442
+ 0.0,
443
+ 0.0,
444
+ 0.0,
445
+ 0.0,
446
+ 0.0,
447
+ 0.0,
448
+ 0.0,
449
+ 0.0,
450
+ 0.0,
451
+ 0.0,
452
+ 0.0,
453
+ 0.0,
454
+ 0.0,
455
+ 0.0,
456
+ 0.0,
457
+ 0.0,
458
+ 0.0,
459
+ 0.0,
460
+ 0.0,
461
+ 0.0,
462
+ 0.0,
463
+ 0.0,
464
+ 0.0,
465
+ 0.0,
466
+ 0.0,
467
+ 0.0,
468
+ 0.0,
469
+ 0.0,
470
+ 0.0,
471
+ 0.0,
472
+ 0.0,
473
+ 0.0,
474
+ 0.0,
475
+ 0.0,
476
+ 0.0,
477
+ 0.0,
478
+ 0.0,
479
+ 0.0,
480
+ 0.0,
481
+ 0.0,
482
+ 0.0,
483
+ 0.0,
484
+ 0.0,
485
+ 0.0,
486
+ 0.0,
487
+ 0.0,
488
+ 0.0,
489
+ 0.0,
490
+ 0.0,
491
+ 0.0,
492
+ 0.0,
493
+ 0.0,
494
+ 0.0,
495
+ 0.0,
496
+ 0.0,
497
+ 0.0,
498
+ 0.0,
499
+ 0.0,
500
+ 0.0,
501
+ 0.0,
502
+ 0.0,
503
+ 0.0,
504
+ 0.0,
505
+ 0.0,
506
+ 0.0,
507
+ 0.0,
508
+ 0.0,
509
+ 0.0,
510
+ 0.0,
511
+ 0.0,
512
+ 0.0,
513
+ 0.0,
514
+ 0.0,
515
+ 0.0,
516
+ 0.0,
517
+ 0.0,
518
+ 0.0,
519
+ 0.0
520
+ ],
521
+ "eos_token_id": 10,
522
+ "head_dim": 128,
523
+ "hidden_act": "silu",
524
+ "hidden_size": 640,
525
+ "initializer_range": 0.02,
526
+ "intermediate_size": 2560,
527
+ "max_position_embeddings": 32770,
528
+ "model_type": "mistral",
529
+ "num_attention_heads": 32,
530
+ "num_hidden_layers": 23,
531
+ "num_key_value_heads": 8,
532
+ "pad_token_id": 9,
533
+ "rms_norm_eps": 1e-05,
534
+ "rope_theta": 100000,
535
+ "sep_token_id": 7,
536
+ "sliding_window": null,
537
+ "tie_word_embeddings": false,
538
+ "tokenizer_config": {
539
+ "char_ords": [
540
+ 68,
541
+ 78,
542
+ 65,
543
+ 84,
544
+ 71,
545
+ 67
546
+ ],
547
+ "model_max_length": 6144
548
+ },
549
+ "torch_dtype": "bfloat16",
550
+ "transformers_version": "4.50.0",
551
+ "use_cache": false,
552
+ "vocab_size": 512
553
+ }
figures/biotoken_biofm.png ADDED

Git LFS Details

  • SHA256: 8e74f2f354ba8b915bf7701e916cdadb5ab676cdef42552b3871c9e8d3243bdf
  • Pointer size: 131 Bytes
  • Size of remote file: 644 kB
figures/expression_model_comparison.png ADDED

Git LFS Details

  • SHA256: e34f37a6cbd24bb7ec4d1f648e8b34715478da0fe32ae63bf2efca84efb21b55
  • Pointer size: 131 Bytes
  • Size of remote file: 272 kB
figures/nt_lr_heatmap_and_barh.png ADDED

Git LFS Details

  • SHA256: 7944ff636dec1c93284720874763e11acd65cd202a02aac57270c6d57a2b9bc7
  • Pointer size: 131 Bytes
  • Size of remote file: 302 kB
figures/sqtl_model_comparison.png ADDED

Git LFS Details

  • SHA256: c1311152e49545ec71b98d18412598896f886d83139b0e5b325665d724be14e6
  • Pointer size: 131 Bytes
  • Size of remote file: 294 kB
figures/vb_heatmap_and_barh_max.png ADDED

Git LFS Details

  • SHA256: ea2eb5a16d0d983bd9a141317b93c4a1ae8239ed858a0b430475d2adf26dcc0e
  • Pointer size: 131 Bytes
  • Size of remote file: 514 kB
generation_config.json ADDED
@@ -0,0 +1,8 @@
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "_from_model_config": true,
3
+ "bos_token_id": 11,
4
+ "eos_token_id": 10,
5
+ "pad_token_id": 9,
6
+ "transformers_version": "4.50.0",
7
+ "use_cache": false
8
+ }
model.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:ff6ea18ca716b5686ad3e0f025e0d3c86257a88766b6aa0063db5cb6493ae108
3
+ size 528959288
tokenizer_config.json ADDED
@@ -0,0 +1,11 @@
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "char_ords": [
3
+ 68,
4
+ 78,
5
+ 65,
6
+ 84,
7
+ 71,
8
+ 67
9
+ ],
10
+ "model_max_length": 6144
11
+ }