TinyCLIP-39M-PixelArt

This is a Sentence Transformers model fine-tuned from TinyCLIP-ViT-39M on Angais/pixel-art-atlas-1.4m for finding pixel-art images from short English descriptions. Both the text and image encoders were trained. It maps text and images to a shared 512-dimensional space, with Matryoshka support at 256 and 128 dimensions.

I also exported only the text-tower from this model into tomaarsen/TinyCLIP-39M-PixelArt-Text, so you don't need to load the image tower into memory when processing queries.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: wkcn/TinyCLIP-ViT-39M-16-Text-19M-YFCC15M
• Maximum Sequence Length: 77 text tokens
• Output Dimensionality: 512 dimensions
• Similarity Function: Cosine Similarity
• Supported Modalities: Text, Image
• Training Dataset: Angais/pixel-art-atlas-1.4m
• License: Apache 2.0

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'transformer_task': 'feature-extraction', 'modality_config': {'text': {'method': 'get_text_features', 'method_output_name': 'pooler_output'}, 'image': {'method': 'get_image_features', 'method_output_name': 'pooler_output'}}, 'module_output_name': 'sentence_embedding', 'architecture': 'CLIPModel'})
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("tomaarsen/TinyCLIP-39M-PixelArt")
# Run inference
queries = [
    'sturdy brass hook',
]
documents = [
    'https://huggingface.co/tomaarsen/TinyCLIP-39M-PixelArt/resolve/main/assets/image_0.jpg',
    'https://huggingface.co/tomaarsen/TinyCLIP-39M-PixelArt/resolve/main/assets/image_1.jpg',
    'https://huggingface.co/tomaarsen/TinyCLIP-39M-PixelArt/resolve/main/assets/image_2.jpg',
]
query_embeddings = model.encode_query(queries)
document_embeddings = model.encode_document(documents)
print(query_embeddings.shape, document_embeddings.shape)
# [1, 512] [3, 512]

# Get the similarity scores for the embeddings
similarities = model.similarity(query_embeddings, document_embeddings)
print(similarities)
# tensor([[ 0.5635, -0.0755, -0.1246]])

Pass truncate_dim=256 or truncate_dim=128 when loading the model to use smaller embeddings. Use the same dimension for queries and images. Set normalize_embeddings=True when encoding for a dot-product search index.

The example images are RGB. For your own transparent pixel art, composite the image onto an RGB (128, 128, 128) background before encoding, matching the training preprocessing.

Training Details

Training Dataset

pixel-art-atlas-1.4m

• Dataset: Angais/pixel-art-atlas-1.4m
• Size: 2,753,652 training samples
• Columns: query and image
• Approximate statistics based on the first 100 samples:

  	query	image
  type	string	image
  modality	text	image
  details	min: 3 tokens mean: 4.62 tokens max: 9 tokens	min: 16x16 px mean: 16x16 px max: 16x16 px

• Samples:

  query	image
  sturdy brass hook
  hook
  chicken drumstick

• Loss: MatryoshkaLoss with these parameters:

  {
      "loss": "MultipleNegativesRankingLoss",
      "matryoshka_dims": [
          512,
          256,
          128
      ],
      "matryoshka_weights": [
          1,
          1,
          1
      ],
      "n_dims_per_step": -1
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• per_device_train_batch_size: 256
• num_train_epochs: 1
• learning_rate: 2e-05
• lr_scheduler_type: cosine
• warmup_steps: 0.05
• bf16: True
• disable_tqdm: True
• data_seed: 42
• dataloader_drop_last: True

All Hyperparameters

Click to expand

• per_device_train_batch_size: 256
• num_train_epochs: 1
• max_steps: -1
• learning_rate: 2e-05
• lr_scheduler_type: cosine
• lr_scheduler_kwargs: None
• warmup_steps: 0.05
• optim: adamw_torch_fused
• optim_args: None
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• optim_target_modules: None
• gradient_accumulation_steps: 1
• average_tokens_across_devices: True
• max_grad_norm: 1.0
• label_smoothing_factor: 0.0
• bf16: True
• fp16: False
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• use_liger_kernel: False
• liger_kernel_config: None
• use_cache: False
• neftune_noise_alpha: None
• torch_empty_cache_steps: None
• auto_find_batch_size: False
• log_on_each_node: True
• logging_nan_inf_filter: True
• include_num_input_tokens_seen: no
• log_level: passive
• log_level_replica: warning
• disable_tqdm: True
• project: huggingface
• trackio_space_id: None
• trackio_bucket_id: None
• trackio_static_space_id: None
• per_device_eval_batch_size: 8
• prediction_loss_only: True
• eval_on_start: False
• eval_do_concat_batches: True
• eval_use_gather_object: False
• eval_accumulation_steps: None
• include_for_metrics: []
• batch_eval_metrics: False
• save_only_model: False
• save_on_each_node: False
• enable_jit_checkpoint: False
• push_to_hub: False
• hub_private_repo: None
• hub_model_id: None
• hub_strategy: every_save
• hub_always_push: False
• hub_revision: None
• load_best_model_at_end: False
• ignore_data_skip: False
• restore_callback_states_from_checkpoint: False
• full_determinism: False
• seed: 42
• data_seed: 42
• use_cpu: False
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• parallelism_config: None
• dataloader_drop_last: True
• dataloader_num_workers: 0
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• dataloader_prefetch_factor: None
• dataloader_multiprocessing_context: None
• dataloader_in_order: True
• remove_unused_columns: True
• label_names: None
• train_sampling_strategy: random
• length_column_name: length
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• ddp_static_graph: None
• ddp_backend: None
• ddp_timeout: 1800
• fsdp: None
• fsdp_config: None
• deepspeed: None
• debug: []
• skip_memory_metrics: True
• do_predict: False
• resume_from_checkpoint: None
• local_rank: -1
• prompts: None
• multi_dataset_batch_sampler: proportional
• router_mapping: {}
• learning_rate_mapping: {}
• activation_checkpointing: False
• activation_checkpointing_kwargs: None
• warmup_ratio: None

Training Logs

Click to expand

Epoch	Step	Training Loss
0.0009	10	11.7114
0.0019	20	11.1023
0.0028	30	10.0764
0.0037	40	8.6836
0.0046	50	7.6932
0.0056	60	6.5590
0.0065	70	5.8142
0.0074	80	5.2593
0.0084	90	4.6366
0.0093	100	4.4097
0.0102	110	4.2301
0.0112	120	4.0108
0.0121	130	3.6869
0.0130	140	3.4337
0.0139	150	3.3695
0.0149	160	3.1533
0.0158	170	3.1227
0.0167	180	2.9738
0.0177	190	2.8259
0.0186	200	2.7140
0.0195	210	2.6136
0.0205	220	2.7021
0.0214	230	2.5875
0.0223	240	2.4097
0.0232	250	2.3561
0.0242	260	2.3429
0.0251	270	2.1613
0.0260	280	2.0224
0.0270	290	2.2490
0.0279	300	2.0222
0.0288	310	2.0288
0.0298	320	2.0441
0.0307	330	2.0224
0.0316	340	1.9492
0.0325	350	1.9792
0.0335	360	1.9611
0.0344	370	1.7949
0.0353	380	1.8400
0.0363	390	1.8321
0.0372	400	1.7703
0.0381	410	1.7357
0.0390	420	1.7170
0.0400	430	1.7542
0.0409	440	1.6532
0.0418	450	1.6008
0.0428	460	1.6439
0.0437	470	1.6417
0.0446	480	1.5966
0.0456	490	1.5689
0.0465	500	1.5090
0.0474	510	1.5493
0.0483	520	1.4916
0.0493	530	1.4202
0.0502	540	1.4658
0.0511	550	1.4453
0.0521	560	1.4557
0.0530	570	1.5424
0.0539	580	1.4643
0.0549	590	1.3645
0.0558	600	1.4451
0.0567	610	1.3476
0.0576	620	1.4512
0.0586	630	1.4555
0.0595	640	1.2428
0.0604	650	1.2103
0.0614	660	1.3732
0.0623	670	1.3427
0.0632	680	1.4099
0.0642	690	1.2969
0.0651	700	1.2580
0.0660	710	1.2359
0.0669	720	1.2203
0.0679	730	1.2321
0.0688	740	1.2167
0.0697	750	1.3023
0.0707	760	1.2736
0.0716	770	1.1570
0.0725	780	1.2331
0.0734	790	1.1464
0.0744	800	1.2717
0.0753	810	1.1370
0.0762	820	1.2091
0.0772	830	1.2404
0.0781	840	1.1254
0.0790	850	1.1224
0.0800	860	1.1088
0.0809	870	1.2243
0.0818	880	1.1837
0.0827	890	1.2300
0.0837	900	1.1474
0.0846	910	1.1493
0.0855	920	1.1777
0.0865	930	1.1099
0.0874	940	1.0691
0.0883	950	1.1780
0.0893	960	1.0774
0.0902	970	1.0989
0.0911	980	1.1114
0.0920	990	1.1377
0.0930	1000	1.0267
0.0939	1010	1.0521
0.0948	1020	1.0994
0.0958	1030	1.0497
0.0967	1040	1.0416
0.0976	1050	1.0778
0.0985	1060	0.9890
0.0995	1070	1.0693
0.1004	1080	0.9598
0.1013	1090	0.9871
0.1023	1100	1.1215
0.1032	1110	0.9875
0.1041	1120	0.9940
0.1051	1130	0.9889
0.1060	1140	0.9859
0.1069	1150	1.0096
0.1078	1160	1.0121
0.1088	1170	0.9998
0.1097	1180	0.9232
0.1106	1190	0.9525
0.1116	1200	1.0372
0.1125	1210	0.9496
0.1134	1220	0.9686
0.1144	1230	0.9433
0.1153	1240	0.9392
0.1162	1250	0.9186
0.1171	1260	0.8986
0.1181	1270	0.9252
0.1190	1280	0.9585
0.1199	1290	0.8803
0.1209	1300	0.8638
0.1218	1310	0.8824
0.1227	1320	0.9649
0.1237	1330	0.8740
0.1246	1340	0.9503
0.1255	1350	1.0364
0.1264	1360	0.8567
0.1274	1370	0.8539
0.1283	1380	0.9036
0.1292	1390	0.8880
0.1302	1400	0.8631
0.1311	1410	0.8994
0.1320	1420	0.8695
0.1329	1430	0.8638
0.1339	1440	0.9380
0.1348	1450	0.9090
0.1357	1460	0.8953
0.1367	1470	0.9201
0.1376	1480	0.8478
0.1385	1490	0.8611
0.1395	1500	0.8670
0.1404	1510	0.8558
0.1413	1520	0.8849
0.1422	1530	0.7945
0.1432	1540	0.8272
0.1441	1550	0.8658
0.1450	1560	0.9137
0.1460	1570	0.8985
0.1469	1580	0.7985
0.1478	1590	0.8142
0.1488	1600	0.8704
0.1497	1610	0.9078
0.1506	1620	0.9363
0.1515	1630	0.8869
0.1525	1640	0.7946
0.1534	1650	0.7857
0.1543	1660	0.9412
0.1553	1670	0.8777
0.1562	1680	0.8582
0.1571	1690	0.8986
0.1581	1700	0.8174
0.1590	1710	0.8818
0.1599	1720	0.8176
0.1608	1730	0.7711
0.1618	1740	0.8130
0.1627	1750	0.7700
0.1636	1760	0.8833
0.1646	1770	0.7642
0.1655	1780	0.8094
0.1664	1790	0.8028
0.1673	1800	0.8791
0.1683	1810	0.7685
0.1692	1820	0.7408
0.1701	1830	0.7834
0.1711	1840	0.8233
0.1720	1850	0.8205
0.1729	1860	0.8247
0.1739	1870	0.7627
0.1748	1880	0.7437
0.1757	1890	0.8181
0.1766	1900	0.7871
0.1776	1910	0.8252
0.1785	1920	0.7756
0.1794	1930	0.7789
0.1804	1940	0.7810
0.1813	1950	0.9078
0.1822	1960	0.7774
0.1832	1970	0.7307
0.1841	1980	0.7594
0.1850	1990	0.7712
0.1859	2000	0.8255
0.1869	2010	0.6801
0.1878	2020	0.7632
0.1887	2030	0.8033
0.1897	2040	0.8275
0.1906	2050	0.7975
0.1915	2060	0.7688
0.1925	2070	0.7396
0.1934	2080	0.6973
0.1943	2090	0.7100
0.1952	2100	0.7316
0.1962	2110	0.7825
0.1971	2120	0.6881
0.1980	2130	0.7676
0.1990	2140	0.7396
0.1999	2150	0.7602
0.2008	2160	0.7271
0.2017	2170	0.7915
0.2027	2180	0.7487
0.2036	2190	0.7802
0.2045	2200	0.7031
0.2055	2210	0.6641
0.2064	2220	0.6803
0.2073	2230	0.7063
0.2083	2240	0.7378
0.2092	2250	0.7078
0.2101	2260	0.7101
0.2110	2270	0.7430
0.2120	2280	0.6405
0.2129	2290	0.7610
0.2138	2300	0.7124
0.2148	2310	0.7093
0.2157	2320	0.6463
0.2166	2330	0.7249
0.2176	2340	0.7157
0.2185	2350	0.7288
0.2194	2360	0.6942
0.2203	2370	0.7335
0.2213	2380	0.7086
0.2222	2390	0.7370
0.2231	2400	0.6741
0.2241	2410	0.7711
0.2250	2420	0.7138
0.2259	2430	0.6883
0.2269	2440	0.7160
0.2278	2450	0.7158
0.2287	2460	0.7032
0.2296	2470	0.6716
0.2306	2480	0.7020
0.2315	2490	0.6698
0.2324	2500	0.6782
0.2334	2510	0.6989
0.2343	2520	0.6645
0.2352	2530	0.7445
0.2361	2540	0.6600
0.2371	2550	0.7347
0.2380	2560	0.6680
0.2389	2570	0.6984
0.2399	2580	0.6269
0.2408	2590	0.7409
0.2417	2600	0.6879
0.2427	2610	0.7088
0.2436	2620	0.6867
0.2445	2630	0.6400
0.2454	2640	0.6875
0.2464	2650	0.6944
0.2473	2660	0.7516
0.2482	2670	0.7199
0.2492	2680	0.7206
0.2501	2690	0.6705
0.2510	2700	0.7218
0.2520	2710	0.6621
0.2529	2720	0.6757
0.2538	2730	0.6988
0.2547	2740	0.6192
0.2557	2750	0.6362
0.2566	2760	0.6539
0.2575	2770	0.6212
0.2585	2780	0.6816
0.2594	2790	0.6033
0.2603	2800	0.6371
0.2612	2810	0.6009
0.2622	2820	0.6596
0.2631	2830	0.6628
0.2640	2840	0.6050
0.2650	2850	0.6491
0.2659	2860	0.6617
0.2668	2870	0.6518
0.2678	2880	0.6861
0.2687	2890	0.6554
0.2696	2900	0.6870
0.2705	2910	0.6022
0.2715	2920	0.6268
0.2724	2930	0.5884
0.2733	2940	0.6518
0.2743	2950	0.6170
0.2752	2960	0.6779
0.2761	2970	0.6774
0.2771	2980	0.5925
0.2780	2990	0.6539
0.2789	3000	0.6148
0.2798	3010	0.6020
0.2808	3020	0.7208
0.2817	3030	0.5805
0.2826	3040	0.6507
0.2836	3050	0.6110
0.2845	3060	0.7345
0.2854	3070	0.6110
0.2864	3080	0.5654
0.2873	3090	0.6347
0.2882	3100	0.6979
0.2891	3110	0.6315
0.2901	3120	0.6629
0.2910	3130	0.6583
0.2919	3140	0.6874
0.2929	3150	0.6170
0.2938	3160	0.6295
0.2947	3170	0.6094
0.2956	3180	0.6217
0.2966	3190	0.6447
0.2975	3200	0.6140
0.2984	3210	0.6549
0.2994	3220	0.6147
0.3003	3230	0.5976
0.3012	3240	0.5836
0.3022	3250	0.6052
0.3031	3260	0.6319
0.3040	3270	0.5895
0.3049	3280	0.5322
0.3059	3290	0.5598
0.3068	3300	0.5670
0.3077	3310	0.6367
0.3087	3320	0.6115
0.3096	3330	0.6072
0.3105	3340	0.5961
0.3115	3350	0.6660
0.3124	3360	0.6580
0.3133	3370	0.5647
0.3142	3380	0.5890
0.3152	3390	0.5686
0.3161	3400	0.6202
0.3170	3410	0.5702
0.3180	3420	0.5691
0.3189	3430	0.5686
0.3198	3440	0.5970
0.3208	3450	0.5297
0.3217	3460	0.5497
0.3226	3470	0.5469
0.3235	3480	0.5443
0.3245	3490	0.5926
0.3254	3500	0.5782
0.3263	3510	0.5883
0.3273	3520	0.5486
0.3282	3530	0.5526
0.3291	3540	0.5792
0.3300	3550	0.5587
0.3310	3560	0.6082
0.3319	3570	0.6286
0.3328	3580	0.6076
0.3338	3590	0.5788
0.3347	3600	0.6050
0.3356	3610	0.5891
0.3366	3620	0.6008
0.3375	3630	0.5152
0.3384	3640	0.6072
0.3393	3650	0.6413
0.3403	3660	0.5881
0.3412	3670	0.5677
0.3421	3680	0.5785
0.3431	3690	0.5662
0.3440	3700	0.5352
0.3449	3710	0.5601
0.3459	3720	0.5981
0.3468	3730	0.5946
0.3477	3740	0.5590
0.3486	3750	0.5756
0.3496	3760	0.5519
0.3505	3770	0.5729
0.3514	3780	0.5848
0.3524	3790	0.5763
0.3533	3800	0.5714
0.3542	3810	0.5431
0.3552	3820	0.5584
0.3561	3830	0.5439
0.3570	3840	0.5493
0.3579	3850	0.5788
0.3589	3860	0.5088
0.3598	3870	0.5860
0.3607	3880	0.5468
0.3617	3890	0.5458
0.3626	3900	0.5766
0.3635	3910	0.5771
0.3644	3920	0.6084
0.3654	3930	0.5790
0.3663	3940	0.5630
0.3672	3950	0.4968
0.3682	3960	0.5120
0.3691	3970	0.5478
0.3700	3980	0.5426
0.3710	3990	0.5510
0.3719	4000	0.6309
0.3728	4010	0.5656
0.3737	4020	0.4827
0.3747	4030	0.4980
0.3756	4040	0.5807
0.3765	4050	0.5169
0.3775	4060	0.5240
0.3784	4070	0.4822
0.3793	4080	0.5358
0.3803	4090	0.5715
0.3812	4100	0.5445
0.3821	4110	0.5018
0.3830	4120	0.5063
0.3840	4130	0.5715
0.3849	4140	0.4941
0.3858	4150	0.5619
0.3868	4160	0.5501
0.3877	4170	0.5163
0.3886	4180	0.4902
0.3896	4190	0.5314
0.3905	4200	0.5412
0.3914	4210	0.5366
0.3923	4220	0.5278
0.3933	4230	0.5226
0.3942	4240	0.5205
0.3951	4250	0.5553
0.3961	4260	0.6160
0.3970	4270	0.5310
0.3979	4280	0.5292
0.3988	4290	0.5133
0.3998	4300	0.5279
0.4007	4310	0.4956
0.4016	4320	0.5297
0.4026	4330	0.5634
0.4035	4340	0.5627
0.4044	4350	0.5212
0.4054	4360	0.5666
0.4063	4370	0.5310
0.4072	4380	0.5270
0.4081	4390	0.5072
0.4091	4400	0.4998
0.4100	4410	0.5493
0.4109	4420	0.4814
0.4119	4430	0.5360
0.4128	4440	0.5510
0.4137	4450	0.5435
0.4147	4460	0.5073
0.4156	4470	0.5523
0.4165	4480	0.5110
0.4174	4490	0.5045
0.4184	4500	0.5767
0.4193	4510	0.4948
0.4202	4520	0.5289
0.4212	4530	0.5331
0.4221	4540	0.5012
0.4230	4550	0.5014
0.4239	4560	0.5605
0.4249	4570	0.5127
0.4258	4580	0.4562
0.4267	4590	0.4898
0.4277	4600	0.4287
0.4286	4610	0.4934
0.4295	4620	0.5198
0.4305	4630	0.5446
0.4314	4640	0.5813
0.4323	4650	0.5177
0.4332	4660	0.5778
0.4342	4670	0.4944
0.4351	4680	0.5225
0.4360	4690	0.4880
0.4370	4700	0.5177
0.4379	4710	0.4643
0.4388	4720	0.4644
0.4398	4730	0.5478
0.4407	4740	0.4993
0.4416	4750	0.6210
0.4425	4760	0.5486
0.4435	4770	0.4971
0.4444	4780	0.4756
0.4453	4790	0.5442
0.4463	4800	0.4955
0.4472	4810	0.5135
0.4481	4820	0.4773
0.4491	4830	0.5474
0.4500	4840	0.4775
0.4509	4850	0.5131
0.4518	4860	0.5344
0.4528	4870	0.4292
0.4537	4880	0.4965
0.4546	4890	0.4470
0.4556	4900	0.5857
0.4565	4910	0.5249
0.4574	4920	0.5269
0.4583	4930	0.4875
0.4593	4940	0.4871
0.4602	4950	0.5356
0.4611	4960	0.4691
0.4621	4970	0.4661
0.4630	4980	0.4510
0.4639	4990	0.4847
0.4649	5000	0.5102
0.4658	5010	0.4388
0.4667	5020	0.4734
0.4676	5030	0.5027
0.4686	5040	0.5079
0.4695	5050	0.4598
0.4704	5060	0.4468
0.4714	5070	0.5195
0.4723	5080	0.4865
0.4732	5090	0.4282
0.4742	5100	0.5058
0.4751	5110	0.4826
0.4760	5120	0.5189
0.4769	5130	0.4816
0.4779	5140	0.4479
0.4788	5150	0.4188
0.4797	5160	0.5150
0.4807	5170	0.4520
0.4816	5180	0.5416
0.4825	5190	0.4697
0.4835	5200	0.4753
0.4844	5210	0.4530
0.4853	5220	0.4649
0.4862	5230	0.4155
0.4872	5240	0.4095
0.4881	5250	0.5159
0.4890	5260	0.4388
0.4900	5270	0.4700
0.4909	5280	0.4491
0.4918	5290	0.4655
0.4927	5300	0.4606
0.4937	5310	0.4726
0.4946	5320	0.5065
0.4955	5330	0.4535
0.4965	5340	0.5337
0.4974	5350	0.4537
0.4983	5360	0.4257
0.4993	5370	0.4754
0.5002	5380	0.4838
0.5011	5390	0.4514
0.5020	5400	0.4913
0.5030	5410	0.4668
0.5039	5420	0.4727
0.5048	5430	0.4605
0.5058	5440	0.4780
0.5067	5450	0.4157
0.5076	5460	0.4370
0.5086	5470	0.5119
0.5095	5480	0.3965
0.5104	5490	0.4515
0.5113	5500	0.4282
0.5123	5510	0.4697
0.5132	5520	0.4481
0.5141	5530	0.4587
0.5151	5540	0.4554
0.5160	5550	0.4734
0.5169	5560	0.4469
0.5179	5570	0.4681
0.5188	5580	0.4271
0.5197	5590	0.4752
0.5206	5600	0.4728
0.5216	5610	0.4236
0.5225	5620	0.5049
0.5234	5630	0.4829
0.5244	5640	0.4366
0.5253	5650	0.4716
0.5262	5660	0.4423
0.5271	5670	0.4853
0.5281	5680	0.3889
0.5290	5690	0.4197
0.5299	5700	0.4823
0.5309	5710	0.4451
0.5318	5720	0.4446
0.5327	5730	0.4431
0.5337	5740	0.4432
0.5346	5750	0.4546
0.5355	5760	0.3789
0.5364	5770	0.4370
0.5374	5780	0.4682
0.5383	5790	0.5205
0.5392	5800	0.4193
0.5402	5810	0.3758
0.5411	5820	0.4797
0.5420	5830	0.4283
0.5430	5840	0.3958
0.5439	5850	0.4021
0.5448	5860	0.4099
0.5457	5870	0.4183
0.5467	5880	0.4314
0.5476	5890	0.4174
0.5485	5900	0.4646
0.5495	5910	0.3982
0.5504	5920	0.4283
0.5513	5930	0.4239
0.5522	5940	0.4200
0.5532	5950	0.4669
0.5541	5960	0.4648
0.5550	5970	0.4762
0.5560	5980	0.4755
0.5569	5990	0.5030
0.5578	6000	0.3831
0.5588	6010	0.4061
0.5597	6020	0.4669
0.5606	6030	0.3756
0.5615	6040	0.4304
0.5625	6050	0.4183
0.5634	6060	0.4655
0.5643	6070	0.4164
0.5653	6080	0.4183
0.5662	6090	0.3893
0.5671	6100	0.4039
0.5681	6110	0.4074
0.5690	6120	0.4180
0.5699	6130	0.4465
0.5708	6140	0.4231
0.5718	6150	0.3994
0.5727	6160	0.4552
0.5736	6170	0.4715
0.5746	6180	0.3572
0.5755	6190	0.4067
0.5764	6200	0.4439
0.5774	6210	0.3784
0.5783	6220	0.3834
0.5792	6230	0.3898
0.5801	6240	0.3425
0.5811	6250	0.4141
0.5820	6260	0.4426
0.5829	6270	0.3965
0.5839	6280	0.4314
0.5848	6290	0.4121
0.5857	6300	0.3743
0.5866	6310	0.3941
0.5876	6320	0.3571
0.5885	6330	0.4145
0.5894	6340	0.4501
0.5904	6350	0.4442
0.5913	6360	0.4170
0.5922	6370	0.4378
0.5932	6380	0.4024
0.5941	6390	0.3902
0.5950	6400	0.4281
0.5959	6410	0.4828
0.5969	6420	0.4524
0.5978	6430	0.4107
0.5987	6440	0.4339
0.5997	6450	0.4325
0.6006	6460	0.3620
0.6015	6470	0.4109
0.6025	6480	0.4133
0.6034	6490	0.4254
0.6043	6500	0.4460
0.6052	6510	0.4343
0.6062	6520	0.3886
0.6071	6530	0.4069
0.6080	6540	0.4350
0.6090	6550	0.3544
0.6099	6560	0.4424
0.6108	6570	0.3608
0.6118	6580	0.4056
0.6127	6590	0.4633
0.6136	6600	0.4144
0.6145	6610	0.3997
0.6155	6620	0.3764
0.6164	6630	0.3762
0.6173	6640	0.3763
0.6183	6650	0.4200
0.6192	6660	0.3897
0.6201	6670	0.3960
0.6210	6680	0.3269
0.6220	6690	0.4502
0.6229	6700	0.4378
0.6238	6710	0.3624
0.6248	6720	0.4108
0.6257	6730	0.4192
0.6266	6740	0.3614
0.6276	6750	0.3857
0.6285	6760	0.3849
0.6294	6770	0.3929
0.6303	6780	0.3615
0.6313	6790	0.4287
0.6322	6800	0.3860
0.6331	6810	0.3933
0.6341	6820	0.4193
0.6350	6830	0.4076
0.6359	6840	0.4094
0.6369	6850	0.3916
0.6378	6860	0.4043
0.6387	6870	0.3994
0.6396	6880	0.4039
0.6406	6890	0.3898
0.6415	6900	0.3378
0.6424	6910	0.3921
0.6434	6920	0.3854
0.6443	6930	0.3495
0.6452	6940	0.3682
0.6462	6950	0.3817
0.6471	6960	0.3698
0.6480	6970	0.3383
0.6489	6980	0.3490
0.6499	6990	0.4011
0.6508	7000	0.3278
0.6517	7010	0.4274
0.6527	7020	0.4074
0.6536	7030	0.3986
0.6545	7040	0.4068
0.6554	7050	0.3781
0.6564	7060	0.4248
0.6573	7070	0.3704
0.6582	7080	0.3964
0.6592	7090	0.4047
0.6601	7100	0.3901
0.6610	7110	0.3634
0.6620	7120	0.4040
0.6629	7130	0.3683
0.6638	7140	0.3455
0.6647	7150	0.4021
0.6657	7160	0.4018
0.6666	7170	0.3763
0.6675	7180	0.3974
0.6685	7190	0.3646
0.6694	7200	0.3881
0.6703	7210	0.3651
0.6713	7220	0.3437
0.6722	7230	0.3911
0.6731	7240	0.3913
0.6740	7250	0.4201
0.6750	7260	0.3646
0.6759	7270	0.3790
0.6768	7280	0.3834
0.6778	7290	0.3747
0.6787	7300	0.3641
0.6796	7310	0.4076
0.6806	7320	0.3757
0.6815	7330	0.3745
0.6824	7340	0.3712
0.6833	7350	0.3716
0.6843	7360	0.4183
0.6852	7370	0.4214
0.6861	7380	0.3153
0.6871	7390	0.3580
0.6880	7400	0.3670
0.6889	7410	0.3769
0.6898	7420	0.3650
0.6908	7430	0.3632
0.6917	7440	0.3974
0.6926	7450	0.3739
0.6936	7460	0.3905
0.6945	7470	0.4088
0.6954	7480	0.3891
0.6964	7490	0.3524
0.6973	7500	0.3756
0.6982	7510	0.3671
0.6991	7520	0.3652
0.7001	7530	0.3480
0.7010	7540	0.3695
0.7019	7550	0.4282
0.7029	7560	0.3207
0.7038	7570	0.3443
0.7047	7580	0.3883
0.7057	7590	0.3728
0.7066	7600	0.3248
0.7075	7610	0.3442
0.7084	7620	0.3418
0.7094	7630	0.3223
0.7103	7640	0.3681
0.7112	7650	0.3877
0.7122	7660	0.3381
0.7131	7670	0.3533
0.7140	7680	0.3910
0.7149	7690	0.3864
0.7159	7700	0.3691
0.7168	7710	0.3734
0.7177	7720	0.3598
0.7187	7730	0.3857
0.7196	7740	0.3819
0.7205	7750	0.3707
0.7215	7760	0.3472
0.7224	7770	0.3463
0.7233	7780	0.4006
0.7242	7790	0.3153
0.7252	7800	0.4091
0.7261	7810	0.3670
0.7270	7820	0.3859
0.7280	7830	0.3524
0.7289	7840	0.3486
0.7298	7850	0.3555
0.7308	7860	0.3821
0.7317	7870	0.3759
0.7326	7880	0.3233
0.7335	7890	0.3702
0.7345	7900	0.3812
0.7354	7910	0.3274
0.7363	7920	0.3549
0.7373	7930	0.3835
0.7382	7940	0.3468
0.7391	7950	0.4166
0.7401	7960	0.3859
0.7410	7970	0.3545
0.7419	7980	0.3344
0.7428	7990	0.3161
0.7438	8000	0.3839
0.7447	8010	0.4179
0.7456	8020	0.3277
0.7466	8030	0.3355
0.7475	8040	0.3620
0.7484	8050	0.3456
0.7493	8060	0.3558
0.7503	8070	0.3582
0.7512	8080	0.3374
0.7521	8090	0.3803
0.7531	8100	0.3648
0.7540	8110	0.3619
0.7549	8120	0.3359
0.7559	8130	0.3610
0.7568	8140	0.3696
0.7577	8150	0.3352
0.7586	8160	0.3209
0.7596	8170	0.3547
0.7605	8180	0.3561
0.7614	8190	0.3750
0.7624	8200	0.3655
0.7633	8210	0.2801
0.7642	8220	0.3151
0.7652	8230	0.3520
0.7661	8240	0.3767
0.7670	8250	0.3922
0.7679	8260	0.3495
0.7689	8270	0.3874
0.7698	8280	0.3487
0.7707	8290	0.3624
0.7717	8300	0.3704
0.7726	8310	0.3508
0.7735	8320	0.4153
0.7745	8330	0.3717
0.7754	8340	0.3326
0.7763	8350	0.4269
0.7772	8360	0.3281
0.7782	8370	0.3256
0.7791	8380	0.3341
0.7800	8390	0.3288
0.7810	8400	0.3504
0.7819	8410	0.3530
0.7828	8420	0.3445
0.7837	8430	0.3413
0.7847	8440	0.3390
0.7856	8450	0.3297
0.7865	8460	0.3222
0.7875	8470	0.3792
0.7884	8480	0.3399
0.7893	8490	0.3156
0.7903	8500	0.3163
0.7912	8510	0.3494
0.7921	8520	0.4052
0.7930	8530	0.3426
0.7940	8540	0.3300
0.7949	8550	0.3371
0.7958	8560	0.3662
0.7968	8570	0.3542
0.7977	8580	0.3232
0.7986	8590	0.3464
0.7996	8600	0.3979
0.8005	8610	0.3197
0.8014	8620	0.3101
0.8023	8630	0.3669
0.8033	8640	0.3378
0.8042	8650	0.3346
0.8051	8660	0.3591
0.8061	8670	0.3473
0.8070	8680	0.3629
0.8079	8690	0.3126
0.8089	8700	0.3811
0.8098	8710	0.3426
0.8107	8720	0.3410
0.8116	8730	0.3591
0.8126	8740	0.3662
0.8135	8750	0.3069
0.8144	8760	0.3798
0.8154	8770	0.2830
0.8163	8780	0.3231
0.8172	8790	0.3900
0.8181	8800	0.3420
0.8191	8810	0.3323
0.8200	8820	0.3433
0.8209	8830	0.3144
0.8219	8840	0.2972
0.8228	8850	0.3500
0.8237	8860	0.3039
0.8247	8870	0.2986
0.8256	8880	0.3705
0.8265	8890	0.3332
0.8274	8900	0.3540
0.8284	8910	0.3730
0.8293	8920	0.3320
0.8302	8930	0.3255
0.8312	8940	0.3238
0.8321	8950	0.3432
0.8330	8960	0.3445
0.8340	8970	0.3500
0.8349	8980	0.2995
0.8358	8990	0.3240
0.8367	9000	0.3046
0.8377	9010	0.3572
0.8386	9020	0.3483
0.8395	9030	0.3272
0.8405	9040	0.3678
0.8414	9050	0.3489
0.8423	9060	0.3637
0.8433	9070	0.3338
0.8442	9080	0.3503
0.8451	9090	0.3237
0.8460	9100	0.3258
0.8470	9110	0.3635
0.8479	9120	0.3131
0.8488	9130	0.3215
0.8498	9140	0.3371
0.8507	9150	0.3391
0.8516	9160	0.3380
0.8525	9170	0.3369
0.8535	9180	0.3148
0.8544	9190	0.3406
0.8553	9200	0.3084
0.8563	9210	0.3565
0.8572	9220	0.3399
0.8581	9230	0.3349
0.8591	9240	0.3085
0.8600	9250	0.3039
0.8609	9260	0.2969
0.8618	9270	0.3228
0.8628	9280	0.3180
0.8637	9290	0.3370
0.8646	9300	0.3174
0.8656	9310	0.3302
0.8665	9320	0.3706
0.8674	9330	0.3470
0.8684	9340	0.3613
0.8693	9350	0.3276
0.8702	9360	0.3268
0.8711	9370	0.3436
0.8721	9380	0.3299
0.8730	9390	0.3531
0.8739	9400	0.3164
0.8749	9410	0.3204
0.8758	9420	0.3265
0.8767	9430	0.3209
0.8776	9440	0.3284
0.8786	9450	0.4033
0.8795	9460	0.3279
0.8804	9470	0.3263
0.8814	9480	0.3240
0.8823	9490	0.3350
0.8832	9500	0.3192
0.8842	9510	0.3431
0.8851	9520	0.3544
0.8860	9530	0.3236
0.8869	9540	0.3397
0.8879	9550	0.3412
0.8888	9560	0.3380
0.8897	9570	0.3426
0.8907	9580	0.3032
0.8916	9590	0.3350
0.8925	9600	0.3410
0.8935	9610	0.3171
0.8944	9620	0.3212
0.8953	9630	0.3291
0.8962	9640	0.3430
0.8972	9650	0.3277
0.8981	9660	0.3374
0.8990	9670	0.3473
0.9000	9680	0.3057
0.9009	9690	0.3387
0.9018	9700	0.3579
0.9028	9710	0.3028
0.9037	9720	0.3233
0.9046	9730	0.3363
0.9055	9740	0.3450
0.9065	9750	0.3233
0.9074	9760	0.3244
0.9083	9770	0.3034
0.9093	9780	0.3323
0.9102	9790	0.3471
0.9111	9800	0.3151
0.9120	9810	0.3260
0.9130	9820	0.3624
0.9139	9830	0.2935
0.9148	9840	0.3431
0.9158	9850	0.2882
0.9167	9860	0.3389
0.9176	9870	0.3374
0.9186	9880	0.3351
0.9195	9890	0.3099
0.9204	9900	0.3051
0.9213	9910	0.3327
0.9223	9920	0.2996
0.9232	9930	0.3345
0.9241	9940	0.3441
0.9251	9950	0.3608
0.9260	9960	0.3026
0.9269	9970	0.2998
0.9279	9980	0.2913
0.9288	9990	0.3128
0.9297	10000	0.3521
0.9306	10010	0.3036
0.9316	10020	0.3742
0.9325	10030	0.3404
0.9334	10040	0.2419
0.9344	10050	0.3221
0.9353	10060	0.3069
0.9362	10070	0.3230
0.9372	10080	0.3249
0.9381	10090	0.3251
0.9390	10100	0.3071
0.9399	10110	0.3152
0.9409	10120	0.2989
0.9418	10130	0.3669
0.9427	10140	0.3001
0.9437	10150	0.3185
0.9446	10160	0.2841
0.9455	10170	0.3321
0.9464	10180	0.3646
0.9474	10190	0.3540
0.9483	10200	0.3645
0.9492	10210	0.3191
0.9502	10220	0.3396
0.9511	10230	0.3067
0.9520	10240	0.3007
0.9530	10250	0.3336
0.9539	10260	0.3084
0.9548	10270	0.3757
0.9557	10280	0.3455
0.9567	10290	0.3252
0.9576	10300	0.3527
0.9585	10310	0.3487
0.9595	10320	0.3291
0.9604	10330	0.3376
0.9613	10340	0.2994
0.9623	10350	0.3022
0.9632	10360	0.2962
0.9641	10370	0.3577
0.9650	10380	0.3470
0.9660	10390	0.3204
0.9669	10400	0.3048
0.9678	10410	0.3146
0.9688	10420	0.3422
0.9697	10430	0.3138
0.9706	10440	0.3279
0.9716	10450	0.3222
0.9725	10460	0.3450
0.9734	10470	0.3484
0.9743	10480	0.3213
0.9753	10490	0.3394
0.9762	10500	0.3438
0.9771	10510	0.3864
0.9781	10520	0.3323
0.9790	10530	0.3352
0.9799	10540	0.3460
0.9808	10550	0.3202
0.9818	10560	0.3400
0.9827	10570	0.3249
0.9836	10580	0.3452
0.9846	10590	0.2910
0.9855	10600	0.3525
0.9864	10610	0.3550
0.9874	10620	0.3219
0.9883	10630	0.3269
0.9892	10640	0.3466
0.9901	10650	0.3269
0.9911	10660	0.3205
0.9920	10670	0.3036
0.9929	10680	0.3438
0.9939	10690	0.3347
0.9948	10700	0.3615
0.9957	10710	0.3264
0.9967	10720	0.3440
0.9976	10730	0.3263
0.9985	10740	0.3727
0.9994	10750	0.3432

Training Time

• Training: 1.4 hours

Framework Versions

• Python: 3.11.6
• Sentence Transformers: 6.3.0.dev0
• Transformers: 5.22.1
• PyTorch: 2.10.0+cu128
• Accelerate: 1.14.0
• Datasets: 4.8.4
• Tokenizers: 0.23.2

Additional Resources

• Training and Finetuning Embedding Models with Sentence Transformers: the end-to-end guide for training or finetuning Sentence Transformer models.
• Introduction to Matryoshka Embedding Models: variable-size embeddings that can be truncated with minimal quality loss.
• Binary and Scalar Embedding Quantization for Significantly Faster & Cheaper Retrieval: post-training compression of embedding vectors.
• Multimodal Embedding & Reranker Models with Sentence Transformers: use text, image, audio, and video models through the same API.
• Training and Finetuning Multimodal Embedding & Reranker Models with Sentence Transformers: train multimodal embedding models, with a Visual Document Retrieval walkthrough.

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MatryoshkaLoss

@misc{kusupati2024matryoshka,
    title={Matryoshka Representation Learning},
    author={Aditya Kusupati and Gantavya Bhatt and Aniket Rege and Matthew Wallingford and Aditya Sinha and Vivek Ramanujan and William Howard-Snyder and Kaifeng Chen and Sham Kakade and Prateek Jain and Ali Farhadi},
    year={2024},
    eprint={2205.13147},
    archivePrefix={arXiv},
    primaryClass={cs.LG}
}

MultipleNegativesRankingLoss

@misc{oord2019representationlearningcontrastivepredictive,
      title={Representation Learning with Contrastive Predictive Coding},
      author={Aaron van den Oord and Yazhe Li and Oriol Vinyals},
      year={2019},
      eprint={1807.03748},
      archivePrefix={arXiv},
      primaryClass={cs.LG},
      url={https://arxiv.org/abs/1807.03748},
}