Instructions to use Sky-Kim/com.sky.sentis.pe-core-t16 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- unity-sentis
How to use Sky-Kim/com.sky.sentis.pe-core-t16 with unity-sentis:
string modelName = "[Your model name here].sentis"; Model model = ModelLoader.Load(Application.streamingAssetsPath + "/" + modelName); IWorker engine = WorkerFactory.CreateWorker(BackendType.GPUCompute, model); // Please see provided C# file for more details
- PerceptionEncoder
How to use Sky-Kim/com.sky.sentis.pe-core-t16 with PerceptionEncoder:
# Use any PE model as a vision encoder import core.vision_encoder.pe as pe model = pe.VisionTransformer.from_config("Sky-Kim/com.sky.sentis.pe-core-t16", pretrained=True) - Notebooks
- Google Colab
- Kaggle
com.sky.sentis.pe-core-t16
Meta Perception Encoder PE-Core-T16-384 (CLIP-style image-text model, Apache-2.0) converted to Unity Inference Engine (Sentis) with Float16 weights. Text and image crops map into one 512-d space: the dot product of two embeddings is their cosine similarity, so you can search people or vehicles in a frame by a text description.
Same C# API as the other com.sky.sentis.* image-text packages (PeCoreS16, Siglip2Base16, ...): swap models by
changing the class name.
Files
Models~/pe_core_t16_vision_fp16.sentis # image [1, 3, 384, 384] RGB 0-1 -> embedding [1, 512] (19.7 MB)
Models~/pe_core_t16_text_fp16.sentis # input_ids [1, 32] int -> embedding [1, 512] (127 MB)
Models~/tokenizer.json # OpenAI CLIP byte-level BPE (49408 tokens)
Runtime/PeCoreT16.cs # self-contained Unity Sentis inference
Usage
using SentisModels;
var pe = new PeCoreT16(BackendType.GPUCompute); // falls back to CPU without compute shaders
pe.Load(modelRoot); // folder holding the Models~ files
pe.Warmup();
float[] query = pe.EmbedText("a photo of a person wearing a red jacket"); // blocking, once per query
float[][] crops = await pe.EmbedCropsAsync(frame, personBoxes); // boxes in source pixels, top-left origin
for (int i = 0; i < crops.Length; i++)
Debug.Log($"{i}: {PeCoreT16.Dot(query, crops[i]):F3}");
PeCoreT16.Dim = 512, PeCoreT16.InputSize = 384, PeCoreT16.ContextLength = 32. LastCropsMs is the duration of the
last EmbedCropsAsync call; Canvas is the last vision input; Tokenize(text) returns the ids the text tower sees.
Preprocessing
- Crops: each box is widened by 15% of its width on each side and 5% of its height top and bottom, clamped to the
texture, and letterboxed (aspect kept, centred) into the 384 x 384 input padded with mid-grey (0.5; the clear
colour is converted to linear in Linear colour-space projects so the model sees 0.498 there and 0.502 in Gamma),
drawn on the GPU into an sRGB render texture (
TextureTransform().SetCoordOrigin(CoordOrigin.TopLeft)). One crop is scheduled per frame; outputs are kept withCopyOutputand read back with anAsyncGPUReadbackcallback, then L2-normalised. The model's(x - 0.5) / 0.5normalisation is baked into the vision graph, so the input is plain RGB in [0, 1]. (Upstream PE preprocessing squashes the whole image to 384 x 384 with bilinear resize; the letterbox is this package's crop rule.) - Text: Unity's built-in
Unity.InferenceEngine.Tokenizationparses the CLIPtokenizer.json(lower-case,<|startoftext|>49406 ...<|endoftext|>49407). Ids are truncated to 32 with the end token kept last and zero-padded, exactly like the reference open_clipSimpleTokenizer(context_length=32). The causal text tower pools at the end token. - Similarity: the upstream
logit_scaleis 100 (softmax over100 * dot), if you want probabilities instead of ranking by cosine.
Conversion notes
- Source:
facebook/PE-Core-T16-384checkpoint with theperception_modelsreference code (core.vision_encoder.pe.CLIP). Vision tower (ViT-T/16, width 192, 12 layers, 2D RoPE + absolute position embedding, class token, attention pooling, 512-d projection) and text tower (width 512, 12 layers, 32 tokens, 512-d projection) exported separately withtorch.onnx.export(legacy exporter, opset 17, batch 1, fixed shapes, weights inline), then imported into Sentis 2.6.1, weights quantized withModelQuantizer.QuantizeWeights(Float16)and saved withModelWriter.Save. Compute stays fp32. - Embeddings are not normalised in the graph;
PeCoreT16L2-normalises them.
Measured parity
Same inputs everywhere: 4 letterboxed crops from the police demo recordings and 5 sentences (cosine similarity of the raw embeddings, worst case; max abs difference).
| Edge | Vision | Text |
|---|---|---|
| PyTorch vs ONNX Runtime | 1.000000 (2.1e-6) | 0.9999998 (1.3e-5) |
| ONNX Runtime vs Sentis fp32, CPU | 1.000000 (2.1e-6) | 1.000000 (1.0e-5) |
| ONNX Runtime vs Sentis fp32, GPUCompute | 1.000000 (1.9e-6) | 1.000000 (1.1e-5) |
| ONNX Runtime vs Sentis fp16 (shipped), CPU | 0.999999 (5.7e-4) | 1.000000 (1.2e-3) |
| ONNX Runtime vs Sentis fp16 (shipped), GPUCompute | 0.999999 (5.7e-4) | 1.000000 (1.2e-3) |
| PyTorch vs Sentis fp16, GPUCompute | 0.999999 | 1.000000 |
- Tokenizer: the C# ids match open_clip
SimpleTokenizerexactly on all test sentences (0 mismatches); over-long text ends with the end token. - Whole C# path (GPU letterbox via
EmbedCropsAsync) vs a PIL bilinear letterbox in Python: cosine 0.994 to 0.999 in a Gamma project, 0.980 to 0.998 in a Linear project (Unity filters sRGB textures in linear light there; resampling differences only).
Zero-shot sanity check (Unity, EmbedCropsAsync + EmbedText, Linear project, GPUCompute; CPU gives the same numbers)
| red jacket | lying | white van | standing (dark) | |
|---|---|---|---|---|
| a photo of a person wearing a red jacket | 0.455 | 0.351 | 0.344 | 0.388 |
| a photo of a person lying on the ground | 0.364 | 0.437 | 0.387 | 0.415 |
| a photo of a white car | 0.335 | 0.361 | 0.465 | 0.370 |
All three queries rank the matching crop first, in Linear and Gamma projects. The margin for "lying" over the standing person is small (0.437 vs 0.415): the tiny model is fine for clothing and vehicle colour but weaker on pose. Use it as one signal next to a pose check.
Timing (warm, Apple M5 Max, Unity 6000.3 editor in batch mode, Sentis 2.6.1; varies a few ms between runs)
| GPUCompute | CPU | |
|---|---|---|
EmbedText, per sentence (blocking) |
9 to 13 ms | 32 to 40 ms |
EmbedCropsAsync, one box (schedule to readback) |
10 to 11 ms | 54 to 68 ms |
EmbedCropsAsync, 4 boxes (one crop per frame) |
31 to 34 ms | 200 to 250 ms |
License & attribution
Model weights: Apache-2.0, Meta Perception Encoder (model card,
code). Paper: Bolya et al., Perception Encoder: The best
visual embeddings are not at the output of the network, 2025 (arXiv:2504.13181).
Tokenizer: OpenAI CLIP (MIT). Wrapper code: MIT, Copyright (c) 2026 Sky Kim. See LICENSE.md.
- Downloads last month
- 14
Model tree for Sky-Kim/com.sky.sentis.pe-core-t16
Base model
facebook/PE-Core-T16-384