com.sky.sentis.pe-core-t16

Meta Perception Encoder PE-Core-T16-384 (CLIP-style image-text model, Apache-2.0) converted to Unity Inference Engine (Sentis) with Float16 weights. Text and image crops map into one 512-d space: the dot product of two embeddings is their cosine similarity, so you can search people or vehicles in a frame by a text description.

Same C# API as the other com.sky.sentis.* image-text packages (PeCoreS16, Siglip2Base16, ...): swap models by changing the class name.

Files

Models~/pe_core_t16_vision_fp16.sentis   # image [1, 3, 384, 384] RGB 0-1 -> embedding [1, 512]   (19.7 MB)
Models~/pe_core_t16_text_fp16.sentis     # input_ids [1, 32] int     -> embedding [1, 512]   (127 MB)
Models~/tokenizer.json                   # OpenAI CLIP byte-level BPE (49408 tokens)
Runtime/PeCoreT16.cs                     # self-contained Unity Sentis inference

Usage

using SentisModels;

var pe = new PeCoreT16(BackendType.GPUCompute);   // falls back to CPU without compute shaders
pe.Load(modelRoot);                                // folder holding the Models~ files
pe.Warmup();

float[] query = pe.EmbedText("a photo of a person wearing a red jacket");   // blocking, once per query
float[][] crops = await pe.EmbedCropsAsync(frame, personBoxes);           // boxes in source pixels, top-left origin
for (int i = 0; i < crops.Length; i++)
    Debug.Log($"{i}: {PeCoreT16.Dot(query, crops[i]):F3}");

PeCoreT16.Dim = 512, PeCoreT16.InputSize = 384, PeCoreT16.ContextLength = 32. LastCropsMs is the duration of the last EmbedCropsAsync call; Canvas is the last vision input; Tokenize(text) returns the ids the text tower sees.

Preprocessing

  • Crops: each box is widened by 15% of its width on each side and 5% of its height top and bottom, clamped to the texture, and letterboxed (aspect kept, centred) into the 384 x 384 input padded with mid-grey (0.5; the clear colour is converted to linear in Linear colour-space projects so the model sees 0.498 there and 0.502 in Gamma), drawn on the GPU into an sRGB render texture (TextureTransform().SetCoordOrigin(CoordOrigin.TopLeft)). One crop is scheduled per frame; outputs are kept with CopyOutput and read back with an AsyncGPUReadback callback, then L2-normalised. The model's (x - 0.5) / 0.5 normalisation is baked into the vision graph, so the input is plain RGB in [0, 1]. (Upstream PE preprocessing squashes the whole image to 384 x 384 with bilinear resize; the letterbox is this package's crop rule.)
  • Text: Unity's built-in Unity.InferenceEngine.Tokenization parses the CLIP tokenizer.json (lower-case, <|startoftext|> 49406 ... <|endoftext|> 49407). Ids are truncated to 32 with the end token kept last and zero-padded, exactly like the reference open_clip SimpleTokenizer(context_length=32). The causal text tower pools at the end token.
  • Similarity: the upstream logit_scale is 100 (softmax over 100 * dot), if you want probabilities instead of ranking by cosine.

Conversion notes

  • Source: facebook/PE-Core-T16-384 checkpoint with the perception_models reference code (core.vision_encoder.pe.CLIP). Vision tower (ViT-T/16, width 192, 12 layers, 2D RoPE + absolute position embedding, class token, attention pooling, 512-d projection) and text tower (width 512, 12 layers, 32 tokens, 512-d projection) exported separately with torch.onnx.export (legacy exporter, opset 17, batch 1, fixed shapes, weights inline), then imported into Sentis 2.6.1, weights quantized with ModelQuantizer.QuantizeWeights(Float16) and saved with ModelWriter.Save. Compute stays fp32.
  • Embeddings are not normalised in the graph; PeCoreT16 L2-normalises them.

Measured parity

Same inputs everywhere: 4 letterboxed crops from the police demo recordings and 5 sentences (cosine similarity of the raw embeddings, worst case; max abs difference).

Edge Vision Text
PyTorch vs ONNX Runtime 1.000000 (2.1e-6) 0.9999998 (1.3e-5)
ONNX Runtime vs Sentis fp32, CPU 1.000000 (2.1e-6) 1.000000 (1.0e-5)
ONNX Runtime vs Sentis fp32, GPUCompute 1.000000 (1.9e-6) 1.000000 (1.1e-5)
ONNX Runtime vs Sentis fp16 (shipped), CPU 0.999999 (5.7e-4) 1.000000 (1.2e-3)
ONNX Runtime vs Sentis fp16 (shipped), GPUCompute 0.999999 (5.7e-4) 1.000000 (1.2e-3)
PyTorch vs Sentis fp16, GPUCompute 0.999999 1.000000
  • Tokenizer: the C# ids match open_clip SimpleTokenizer exactly on all test sentences (0 mismatches); over-long text ends with the end token.
  • Whole C# path (GPU letterbox via EmbedCropsAsync) vs a PIL bilinear letterbox in Python: cosine 0.994 to 0.999 in a Gamma project, 0.980 to 0.998 in a Linear project (Unity filters sRGB textures in linear light there; resampling differences only).

Zero-shot sanity check (Unity, EmbedCropsAsync + EmbedText, Linear project, GPUCompute; CPU gives the same numbers)

red jacket lying white van standing (dark)
a photo of a person wearing a red jacket 0.455 0.351 0.344 0.388
a photo of a person lying on the ground 0.364 0.437 0.387 0.415
a photo of a white car 0.335 0.361 0.465 0.370

All three queries rank the matching crop first, in Linear and Gamma projects. The margin for "lying" over the standing person is small (0.437 vs 0.415): the tiny model is fine for clothing and vehicle colour but weaker on pose. Use it as one signal next to a pose check.

Timing (warm, Apple M5 Max, Unity 6000.3 editor in batch mode, Sentis 2.6.1; varies a few ms between runs)

GPUCompute CPU
EmbedText, per sentence (blocking) 9 to 13 ms 32 to 40 ms
EmbedCropsAsync, one box (schedule to readback) 10 to 11 ms 54 to 68 ms
EmbedCropsAsync, 4 boxes (one crop per frame) 31 to 34 ms 200 to 250 ms

License & attribution

Model weights: Apache-2.0, Meta Perception Encoder (model card, code). Paper: Bolya et al., Perception Encoder: The best visual embeddings are not at the output of the network, 2025 (arXiv:2504.13181). Tokenizer: OpenAI CLIP (MIT). Wrapper code: MIT, Copyright (c) 2026 Sky Kim. See LICENSE.md.

Downloads last month
14
Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Model tree for Sky-Kim/com.sky.sentis.pe-core-t16

Finetuned
(2)
this model

Paper for Sky-Kim/com.sky.sentis.pe-core-t16