--- datasets: - multimolecule/gencode-human library_name: multimolecule license: agpl-3.0 pipeline_tag: text-generation tags: - Biology - DNA - dna widget: - example_title: tumor protein p53 expected_suffix: CGCTGCTCAGATAGCGATGG output: text: ACTCCCCTGCCCTCAACAAGATGTTTTGCCAACTGGCCAAGACCTGCCCTGTGCAGCTGTGGGTTGATTCCACACCCCCGCCCGGCACCCGCGTCCGCGCCATGGCCATCTACAAGCAGTCACAGCACATGACGGAGGTTGTGAGGCGCTGCCCCCACCATGAGC T T G G G G A G G G G T G T T T T T T pipeline_tag: text-generation sequence_type: DNA task: text-generation text: ACTCCCCTGCCCTCAACAAGATGTTTTGCCAACTGGCCAAGACCTGCCCTGTGCAGCTGTGGGTTGATTCCACACCCCCGCCCGGCACCCGCGTCCGCGCCATGGCCATCTACAAGCAGTCACAGCACATGACGGAGGTTGTGAGGCGCTGCCCCCACCATGAG - example_title: BRCA1 DNA repair associated expected_suffix: TCTTAGAGTGTCCCATCTGG output: text: TCATTGGAACAGAAAGAAATGGATTTATCTGCTCTTCGCGTTGAAGAAGTACAAAATGTCATTAATGCTATGCAGAAAAG G T T T T T T T T T T G A C C C C T pipeline_tag: text-generation sequence_type: DNA task: text-generation text: TCATTGGAACAGAAAGAAATGGATTTATCTGCTCTTCGCGTTGAAGAAGTACAAAATGTCATTAATGCTATGCAGAAAA - example_title: hemoglobin subunit beta expected_suffix: GTGGTGAGGCCCTGGGCAGG output: text: CATTTGCTTCTGACACAACTGTGTTCACTAGCAACCTCAAACAGACACCATGGTGCATCTGACTCCTGAGGAGAAGTCTGCCGTTACTGCCCTGTGGGGCAAGGTGAACGTGGATGAAGTTGT T T T T C pipeline_tag: text-generation sequence_type: DNA task: text-generation text: CATTTGCTTCTGACACAACTGTGTTCACTAGCAACCTCAAACAGACACCATGGTGCATCTGACTCCTGAGGAGAAGTCTGCCGTTACTGCCCTGTGGGGCAAGGTGAACGTGGATGAAGTTG - example_title: CF transmembrane conductance regulator expected_suffix: AAGCATGCCAACTAGAAGAG output: text: ACTTCACTTCTAATGGTGATTATGGGAGAACTGGAGCCTTCAGAGGGTAAAATTAAGCACAGTGGAAGAATTTCATTCTGTTCTCAGTTTTCCTGGATTATGCCTGGCACCATTAAAGAAAATATCATCTTTGGTGTTTCCTATGATGAATATAGATACAGAAGCGTCATCAA A A T T T T T A C C C C C C A G G A pipeline_tag: text-generation sequence_type: DNA task: text-generation text: ACTTCACTTCTAATGGTGATTATGGGAGAACTGGAGCCTTCAGAGGGTAAAATTAAGCACAGTGGAAGAATTTCATTCTGTTCTCAGTTTTCCTGGATTATGCCTGGCACCATTAAAGAAAATATCATCTTTGGTGTTTCCTATGATGAATATAGATACAGAAGCGTCATCA - example_title: telomerase reverse transcriptase expected_suffix: TGCTCCGGGCGGACCCGGGG output: text: CGCGGGGGTGGCCGGGGCCAGGGCTTCCCACGTGCGCAGCAGGACGCAGCGCTGCCTGAAACTCGCGCCGCGAGGAGAGGGCGGGGCCGCGGAAAGGAAGGGGAGGGGCTGGGAGGGCCCGGAGGGGGCTGGGCCGGGGACCCGGGAGGGGTCGGGACGGGGCGGGGTCCGCGCGGAGGAGGCGGAGCTGGAAGGTGAAGGGGCAGGACGGGTGCCCGGGTCCCCAGTCCCTCCGCCACGTGGGAAGCGCGGTCCTGGGCGTCTGTGCCCGCGAATCCACTGGGAGCCCGGCCTGGCCCCGACAGCGCAGCG A A A A A A A A A A G G A A A A T T T pipeline_tag: text-generation sequence_type: DNA task: text-generation text: CGCGGGGGTGGCCGGGGCCAGGGCTTCCCACGTGCGCAGCAGGACGCAGCGCTGCCTGAAACTCGCGCCGCGAGGAGAGGGCGGGGCCGCGGAAAGGAAGGGGAGGGGCTGGGAGGGCCCGGAGGGGGCTGGGCCGGGGACCCGGGAGGGGTCGGGACGGGGCGGGGTCCGCGCGGAGGAGGCGGAGCTGGAAGGTGAAGGGGCAGGACGGGTGCCCGGGTCCCCAGTCCCTCCGCCACGTGGGAAGCGCGGTCCTGGGCGTCTGTGCCCGCGAATCCACTGGGAGCCCGGCCTGGCCCCGACAGCGCAGC - example_title: KRAS proto-oncogene expected_suffix: AATATGATCCAACAATAGAG output: text: GCCTGCTGAAAATGACTGAATATAAACTTGTGGTAGTTGGAGCTGGTGGCGTAGGCAAGAGTGCCTTGACGATACAGCTAATTCAGAATCATTTTGTGGACGC C T T T C T T T T T T T T T T T T T pipeline_tag: text-generation sequence_type: DNA task: text-generation text: GCCTGCTGAAAATGACTGAATATAAACTTGTGGTAGTTGGAGCTGGTGGCGTAGGCAAGAGTGCCTTGACGATACAGCTAATTCAGAATCATTTTGTGGACG - example_title: prion protein (Kanno blood group) expected_suffix: GGAGTGACCTGGGCCTCTGC output: text: ATGGCGAACCTTGGCTGCTGGATGCTGGTTCTCTTTGTGGCCACATC C C C C C C T T T T T T T T T T T T G pipeline_tag: text-generation sequence_type: cDNA task: text-generation text: ATGGCGAACCTTGGCTGCTGGATGCTGGTTCTCTTTGTGGCCACAT - example_title: interleukin 10 expected_suffix: TCCTGACTGGGGTGAGGGCC output: text: ATGCACAGCTCAGCACTGCTCTGTTGCCTGGTCCC C C C C C C C C A A G G G A A G A A pipeline_tag: text-generation sequence_type: cDNA task: text-generation text: ATGCACAGCTCAGCACTGCTCTGTTGCCTGGTCC - example_title: Zaire ebolavirus expected_suffix: ATACAATCCCACAGTGTTAT output: text: AATGTTCAAACACTTTGTGAAGCTCTGTTAGCTGATGGTCTTGCTAAAGCATTTCCTAGCAATATGATGGTAGTCACAGAGCGTGAGCAAAAAGAAAGCTTATTGCATCAAGCATCATGGCACCACACAAGTGATGATTTTGGTGAGCATGCCACAGTTAGAGGGAGTAGCTTTGTAACTGATTTAGAGAAATACAATCTTGCATTTAGATATGAGTTTACAGCACCTTTTATAGAATATTGTAACCGTTGCTATGGTGTTAAGAATGTTTTTAATTGGATGCATTA A A A C C T T T T T T T T T T T T C pipeline_tag: text-generation sequence_type: cDNA task: text-generation text: AATGTTCAAACACTTTGTGAAGCTCTGTTAGCTGATGGTCTTGCTAAAGCATTTCCTAGCAATATGATGGTAGTCACAGAGCGTGAGCAAAAAGAAAGCTTATTGCATCAAGCATCATGGCACCACACAAGTGATGATTTTGGTGAGCATGCCACAGTTAGAGGGAGTAGCTTTGTAACTGATTTAGAGAAATACAATCTTGCATTTAGATATGAGTTTACAGCACCTTTTATAGAATATTGTAACCGTTGCTATGGTGTTAAGAATGTTTTTAATTGGATGCATT - example_title: SARS coronavirus expected_suffix: GCACTTTCGAGTACATATCT output: text: ATGTTTATTTTCTTATTATTTCTTACTCTCACTAGTGGTAGTGACCTTGACCGGTGCACCACTTTTGATGATGTTCAAGCTCCTAATTACACTCAACATACTTCATCTATGAGGGGGGTTTACTATCCTGATGAAATTTTTAGATCAGACACTCTTTATTTAACTCAGGATTTATTTCTTCCATTTTATTCTAATGTTACAGGGTTTCATACTATTAATCATACGTTTGACAACCCTGTCATACCTTTTAAGGATGGTATTTATTTTGCTGCCACAGAGAAATCAAATGTTGTCCGTGGTTGGGTTTTTGGTTCTACCATGAACAACAAGTCACAGTCGGTGATTATTATTAACAATTCTACTAATGTTGTTATACGAGCATGTAACTTTGAATTGTGTGACAACCCTTTCTTTGCTGTTTCTAAACCCATGGGTACACAGACACATACTATGATATTCGATAATGCATTTAAATC T T T T A G G G G G G G G C T T T pipeline_tag: text-generation sequence_type: cDNA task: text-generation text: ATGTTTATTTTCTTATTATTTCTTACTCTCACTAGTGGTAGTGACCTTGACCGGTGCACCACTTTTGATGATGTTCAAGCTCCTAATTACACTCAACATACTTCATCTATGAGGGGGGTTTACTATCCTGATGAAATTTTTAGATCAGACACTCTTTATTTAACTCAGGATTTATTTCTTCCATTTTATTCTAATGTTACAGGGTTTCATACTATTAATCATACGTTTGACAACCCTGTCATACCTTTTAAGGATGGTATTTATTTTGCTGCCACAGAGAAATCAAATGTTGTCCGTGGTTGGGTTTTTGGTTCTACCATGAACAACAAGTCACAGTCGGTGATTATTATTAACAATTCTACTAATGTTGTTATACGAGCATGTAACTTTGAATTGTGTGACAACCCTTTCTTTGCTGTTTCTAAACCCATGGGTACACAGACACATACTATGATATTCGATAATGCATTTAAAT - example_title: insulin expected_suffix: TGGAGAACTACTGCAACTAG output: text: ATGGCCCTGTGGATGCGCCTCCTGCCCCTGCTGGCGCTGCTGGCCCTCTGGGGACCTGACCCAGCCGCAGCCTTTGTGAACCAACACCTGTGCGGCTCACACCTGGTGGAAGCTCTCTACCTAGTGTGCGGGGAACGAGGCTTCTTCTACACACCCAAGACCCGCCGGGAGGCAGAGGACCTGCAGGTGGGGCAGGTGGAGCTGGGCGGGGGCCCTGGTGCAGGCAGCCTGCAGCCCTTGGCCCTGGAGGGGTCCCTGCAGAAGCGTGGCATTGTGGAACAATGCTGTACCAGCATCTGCTCCCTCTACCAGCA T T G G T T T T T T T T T T T A C T pipeline_tag: text-generation sequence_type: cDNA task: text-generation text: ATGGCCCTGTGGATGCGCCTCCTGCCCCTGCTGGCGCTGCTGGCCCTCTGGGGACCTGACCCAGCCGCAGCCTTTGTGAACCAACACCTGTGCGGCTCACACCTGGTGGAAGCTCTCTACCTAGTGTGCGGGGAACGAGGCTTCTTCTACACACCCAAGACCCGCCGGGAGGCAGAGGACCTGCAGGTGGGGCAGGTGGAGCTGGGCGGGGGCCCTGGTGCAGGCAGCCTGCAGCCCTTGGCCCTGGAGGGGTCCCTGCAGAAGCGTGGCATTGTGGAACAATGCTGTACCAGCATCTGCTCCCTCTACCAGC - example_title: cyclin dependent kinase inhibitor 2A expected_suffix: CCTCAGACATCCCCGATTGA output: text: ATGGAGCCGGCGGCGGGGAGCAGCATGGAGCCTTCGGCTGACTGGCTGGCCACGGCCGCGGCCCGGGGTCGGGTAGAGGAGGTGCGGGCGCTGCTGGAGGCGGGGGCGCTGCCCAACGCACCGAATAGTTACGGTCGGAGGCCGATCCAGGTCATGATGATGGGCAGCGCCCGAGTGGCGGAGCTGCTGCTGCTCCACGGCGCGGAGCCCAACTGCGCCGACCCCGCCACTCTCACCCGACCCGTGCACGACGCTGCCCGGGAGGGCTTCCTGGACACGCTGGTGGTGCTGCACCGGGCCGGGGCGCGGCTGGACGTGCGCGATGCCTGGGGCCGTCTGCCCGTGGACCTGGCTGAGGAGCTGGGCCATCGCGATGTCGCACGGTACCTGCGCGCGGCTGCGGGGGGCACCAGAGGCAGTAACCATGCCCGCATAGATGCCGCGGAAGGTCG G A A A A A T T T T T T T T T T T C pipeline_tag: text-generation sequence_type: cDNA task: text-generation text: ATGGAGCCGGCGGCGGGGAGCAGCATGGAGCCTTCGGCTGACTGGCTGGCCACGGCCGCGGCCCGGGGTCGGGTAGAGGAGGTGCGGGCGCTGCTGGAGGCGGGGGCGCTGCCCAACGCACCGAATAGTTACGGTCGGAGGCCGATCCAGGTCATGATGATGGGCAGCGCCCGAGTGGCGGAGCTGCTGCTGCTCCACGGCGCGGAGCCCAACTGCGCCGACCCCGCCACTCTCACCCGACCCGTGCACGACGCTGCCCGGGAGGGCTTCCTGGACACGCTGGTGGTGCTGCACCGGGCCGGGGCGCGGCTGGACGTGCGCGATGCCTGGGGCCGTCTGCCCGTGGACCTGGCTGAGGAGCTGGGCCATCGCGATGTCGCACGGTACCTGCGCGCGGCTGCGGGGGGCACCAGAGGCAGTAACCATGCCCGCATAGATGCCGCGGAAGGTC - example_title: human papillomavirus type 16 E6 expected_suffix: GTAGAGAAACCCAGCTGTAA output: text: ATGCACCAAAAGAGAACTGCAATGTTTCAGGACCCACAGGAGCGACCCAGAAAGTTACCACAGTTATGCACAGAGCTGCAAACAACTATACATGATATAATATTAGAATGTGTGTACTGCAAGCAACAGTTACTGCGACGTGAGGTATATGACTTTGCTTTTCGGGATTTATGCATAGTATATAGAGATGGGAATCCATATGCTGTATGTGATAAATGTTTAAAGTTTTATTCTAAAATTAGTGAGTATAGACATTATTGTTATAGTTTGTATGGAACAACATTAGAACAGCAATACAACAAACCGTTGTGTGATTTGTTAATTAGGTGTATTAACTGTCAAAAGCCACTGTGTCCTGAAGAAAAGCAAAGACATCTGGACAAAAAGCAAAGATTCCATAATATAAGGGGTCGGTGGACCGGTCGATGTATGTCTTGTTGCAGATCATCAAGAACACA C C T A A A A T C C C C C T A C T pipeline_tag: text-generation sequence_type: cDNA task: text-generation text: ATGCACCAAAAGAGAACTGCAATGTTTCAGGACCCACAGGAGCGACCCAGAAAGTTACCACAGTTATGCACAGAGCTGCAAACAACTATACATGATATAATATTAGAATGTGTGTACTGCAAGCAACAGTTACTGCGACGTGAGGTATATGACTTTGCTTTTCGGGATTTATGCATAGTATATAGAGATGGGAATCCATATGCTGTATGTGATAAATGTTTAAAGTTTTATTCTAAAATTAGTGAGTATAGACATTATTGTTATAGTTTGTATGGAACAACATTAGAACAGCAATACAACAAACCGTTGTGTGATTTGTTAATTAGGTGTATTAACTGTCAAAAGCCACTGTGTCCTGAAGAAAAGCAAAGACATCTGGACAAAAAGCAAAGATTCCATAATATAAGGGGTCGGTGGACCGGTCGATGTATGTCTTGTTGCAGATCATCAAGAACAC --- # HyenaDNA Pre-trained model on human reference genome using a causal language modeling (CLM) objective with the Hyena operator. ## Disclaimer This is an UNOFFICIAL implementation of the [HyenaDNA: Long-Range Genomic Sequence Modeling at Single Nucleotide Resolution](https://openreview.net/forum?id=ubzNoJjOKj) by Eric Nguyen, Michael Poli, Marjan Faizi, et al. The OFFICIAL repository of HyenaDNA is at [HazyResearch/hyena-dna](https://github.com/HazyResearch/hyena-dna). > [!TIP] > The MultiMolecule team has confirmed that the provided model and checkpoints are producing the same intermediate representations as the original implementation. **The team releasing HyenaDNA did not write this model card for this model so this model card has been written by the MultiMolecule team.** ## Model Details HyenaDNA is a decoder-only model pre-trained on the human reference genome with single nucleotide tokenization in a self-supervised fashion. This means that the model was trained on the raw nucleotides of DNA sequences only, with an automatic process to generate inputs and labels from those sequences. Please refer to the [Training Details](#training-details) section for more information on the training process. HyenaDNA replaces the attention mechanism with the Hyena operator — a subquadratic sequence mixer based on implicit long convolutions. This enables O(L log L) complexity for sequence modeling, allowing the model to handle context lengths up to 1 million base pairs at single nucleotide resolution. The Hyena operator uses: - **Implicit filters**: MLP-parameterized convolution kernels with learned positional embeddings - **Element-wise gating**: Multiplicative interactions between projected input channels - **FFT convolution**: Efficient computation via the Fast Fourier Transform ### Variants HyenaDNA was pretrained across 5 architectures (Table A.1 in the paper), of which 4 have released checkpoints. We provide one converted checkpoint per architecture, using the longest available context length: - **[multimolecule/hyenadna-large](https://huggingface.co/multimolecule/hyenadna-large)**: The HyenaDNA large model with 8 layers, 256 hidden size, and 1,000,000 context length. - **[multimolecule/hyenadna-medium](https://huggingface.co/multimolecule/hyenadna-medium)**: The HyenaDNA medium model with 4 layers, 256 hidden size, and 32,768 context length. - **[multimolecule/hyenadna-small](https://huggingface.co/multimolecule/hyenadna-small)**: The HyenaDNA small model with 2 layers, 256 hidden size, and 1,024 context length. - **[multimolecule/hyenadna-tiny](https://huggingface.co/multimolecule/hyenadna-tiny)**: The HyenaDNA tiny model with 2 layers, 128 hidden size, and 16,384 context length. > [!NOTE] > The original repository releases 7 checkpoints with inconsistent naming (e.g., "large" and "medium" share the same architecture, and multiple "tiny" variants are identical except for context length). We retain only the best (longest context) checkpoint per architecture and use a consistent size-based naming scheme. The mapping from original to MultiMolecule names is: > > | Original Checkpoint | MultiMolecule Name | Architecture | > | ---------------------------------- | ------------------ | ----------------- | > | `hyenadna-large-1m-seqlen-hf` | `hyenadna-large` | 8 layers, 256 dim | > | `hyenadna-small-32k-seqlen-hf` | `hyenadna-medium` | 4 layers, 256 dim | > | `hyenadna-tiny-1k-seqlen-d256-hf` | `hyenadna-small` | 2 layers, 256 dim | > | `hyenadna-tiny-16k-seqlen-d128-hf` | `hyenadna-tiny` | 2 layers, 128 dim | > > The (4, 128) architecture from the paper has no released checkpoint. The remaining 3 original checkpoints (`hyenadna-medium-450k-seqlen-hf`, `hyenadna-medium-160k-seqlen-hf`, `hyenadna-tiny-1k-seqlen-hf`) share the same architecture as one of the above but with shorter context lengths, and are not provided. ### Model Specification
Variants Num Layers Hidden Size Intermediate Size Num Parameters (M) FLOPs (G) MACs (G) Max Num Tokens
HyenaDNA-large 8 256 1024 6.62 6.69 3.35 1,000,002
HyenaDNA-medium 4 3.34 3.35 1.67 32,770
HyenaDNA-small 2 1.71 1.67 0.84 1,026
HyenaDNA-tiny 128 512 0.45 0.44 0.22 16,386
### Links - **Code**: [multimolecule.hyenadna](https://github.com/DLS5-Omics/multimolecule/tree/master/multimolecule/models/hyenadna) - **Data**: [multimolecule/gencode-human](https://huggingface.co/datasets/multimolecule/gencode-human) - **Paper**: [HyenaDNA: Long-Range Genomic Sequence Modeling at Single Nucleotide Resolution](https://openreview.net/forum?id=ubzNoJjOKj) - **Developed by**: Eric Nguyen, Michael Poli, Marjan Faizi, Armin W. Thomas, Callum Birch-Sykes, Michael Wornow, Aman Patel, Clayton Rabideau, Stefano Massaroli, Yoshua Bengio, Stefano Ermon, Christopher Ré, Stephen A. Baccus - **Model type**: Decoder-only with [Hyena](https://arxiv.org/abs/2302.10866) operator - **Original Repository**: [HazyResearch/hyena-dna](https://github.com/HazyResearch/hyena-dna) ## Usage The model file depends on the [`multimolecule`](https://multimolecule.danling.org) library. You can install it using pip: ```bash pip install multimolecule ``` ### Direct Use #### Text Generation You can use this model directly with a pipeline for text generation: ```python import multimolecule # you must import multimolecule to register models from transformers import pipeline generator = pipeline("text-generation", model="multimolecule/hyenadna-tiny") output = generator("ATCGATCGATCG", max_new_tokens=50) ``` ### Downstream Use #### Extract Features Here is how to use this model to get the features of a given sequence in PyTorch: ```python from multimolecule import DnaTokenizer, HyenaDnaModel tokenizer = DnaTokenizer.from_pretrained("multimolecule/hyenadna-tiny") model = HyenaDnaModel.from_pretrained("multimolecule/hyenadna-tiny") text = "ATCGATCGATCGATCG" input = tokenizer(text, return_tensors="pt") output = model(**input) ``` #### Sequence Classification / Regression > [!NOTE] > This model is not fine-tuned for any specific task. You will need to fine-tune the model on a downstream task to use it for sequence classification or regression. Here is how to use this model as backbone to fine-tune for a sequence-level task in PyTorch: ```python import torch from multimolecule import DnaTokenizer, HyenaDnaForSequencePrediction tokenizer = DnaTokenizer.from_pretrained("multimolecule/hyenadna-tiny") model = HyenaDnaForSequencePrediction.from_pretrained("multimolecule/hyenadna-tiny") text = "ATCGATCGATCGATCG" input = tokenizer(text, return_tensors="pt") label = torch.tensor([1]) output = model(**input, labels=label) ``` #### Token Classification / Regression > [!NOTE] > This model is not fine-tuned for any specific task. You will need to fine-tune the model on a downstream task to use it for token classification or regression. Here is how to use this model as backbone to fine-tune for a nucleotide-level task in PyTorch: ```python import torch from multimolecule import DnaTokenizer, HyenaDnaForTokenPrediction tokenizer = DnaTokenizer.from_pretrained("multimolecule/hyenadna-tiny") model = HyenaDnaForTokenPrediction.from_pretrained("multimolecule/hyenadna-tiny") text = "ATCGATCGATCGATCG" input = tokenizer(text, return_tensors="pt") label = torch.randint(2, (len(text), )) output = model(**input, labels=label) ``` ## Training Details HyenaDNA used Causal Language Modeling (CLM) as the pre-training objective: given a DNA sequence, the model is trained to predict the next nucleotide token autoregressively. ### Training Data The HyenaDNA model was pre-trained on the human reference genome (GRCh38). The training data consists of single nucleotide-level DNA sequences from all human chromosomes. Sequences are tokenized at the individual character level (A, C, G, T, N) without k-mer encoding. The dataset is split into training and test sets by chromosome, with held-out chromosomes used for evaluation. ### Training Procedure #### Preprocessing HyenaDNA used causal language modeling (CLM) as the pre-training objective: given a DNA sequence of length L, the model predicts the next nucleotide at each position, i.e., predicting token x\_{t+1} given x_1, ..., x_t. Single nucleotide tokenization is used with a vocabulary of 12 tokens: A, C, G, T, N, and special tokens (CLS, SEP, BOS, MASK, PAD, RESERVED, UNK). #### Sequence Length Warm-up A key training strategy in HyenaDNA is progressive sequence length warm-up. Training begins with short sequences and gradually increases the context length: 1. Training starts with sequences of length L=64. 2. The sequence length is doubled at each warm-up stage (64 → 128 → 256 → ... → target length). 3. This strategy enables stable training at very long context lengths that would be difficult to train from scratch. #### Pre-training The model was trained on up to 8 NVIDIA A100 (80GB) GPUs. - Batch size: 64 -- 256 - Steps: 10,000 -- 20,000 - Optimizer: AdamW - Learning rate: 1.5e-4 -- 6e-4 - Learning rate scheduler: Cosine - Weight decay: 0.1 ## Citation ```bibtex @inproceedings{nguyen2023hyenadna, title={Hyena{DNA}: Long-Range Genomic Sequence Modeling at Single Nucleotide Resolution}, author={Eric Nguyen and Michael Poli and Marjan Faizi and Armin W Thomas and Michael Wornow and Callum Birch-Sykes and Stefano Massaroli and Aman Patel and Clayton M. Rabideau and Yoshua Bengio and Stefano Ermon and Christopher Re and Stephen Baccus}, booktitle={Thirty-seventh Conference on Neural Information Processing Systems}, year={2023}, url={https://openreview.net/forum?id=ubzNoJjOKj} } ``` > [!NOTE] > The artifacts distributed in this repository are part of the MultiMolecule project. > If MultiMolecule supports your research, please cite the MultiMolecule project as follows: ```bibtex @software{chen_2024_12638419, author = {Chen, Zhiyuan and Zhu, Sophia Y.}, title = {MultiMolecule}, doi = {10.5281/zenodo.12638419}, publisher = {Zenodo}, url = {https://doi.org/10.5281/zenodo.12638419}, year = 2024, month = may, day = 4 } ``` ## Contact Please use GitHub issues of [MultiMolecule](https://github.com/DLS5-Omics/multimolecule/issues) for any questions or comments on the model card. Please contact the authors of the [HyenaDNA paper](https://openreview.net/forum?id=ubzNoJjOKj) for questions or comments on the paper/model. ## License This model implementation is licensed under the [GNU Affero General Public License](license.md). For additional terms and clarifications, please refer to our [License FAQ](license-faq.md). ```spdx SPDX-License-Identifier: AGPL-3.0-or-later ```