Bobcat 1.3

Bobcat 1.3 · Bobcat 1.3 FP8 · Bobcat 1.2 · Bobcat Flash 1.2 · Code on GitHub

Update (2026-10-14): on the Jev Decision Index 0.3.2, Bobcat 1.3 scores 62.8 on the Full score and is in the #3 group (tied) with Perplexity Decider v1.1 (62.8) and Darwin-27B-ZTC (62.5), fifth by score of 166 entries and above Jev (60.1) on the same cases. The Full score is 80% private tests that the maintainers ran themselves; see "Decision Index 0.3.2 (official)" under Evaluation.

Bobcat is a typed-decision model. You send a state (text or JSON) and questions whose possible answers you define in the request; Bobcat returns a probability for every answer you named, and nothing else. It never generates text: it reads the logits of the offered candidates at the first answer position of one forward pass, and the host builds a closed JSON reply from your own names. An answer can be wrong, but it cannot be malformed.

This repository, sanghwa-na/bobcat-1.3, holds Bobcat 1.3 as ready-to-serve BF16 weights on google/gemma-4-31B-it (Apache-2.0) at revision 842da3794eaa0b77d5f08bae87a17459d91ff475. It keeps Bobcat 1.2's product behaviour and adds broader decision skills (tool use, claims against evidence, contracts, science and commonsense questions, stance, humor) learned from licensed, human-written public data. vLLM serves it in FP8, and it reads inputs of up to 98,368 tokens without truncating them.

Highlights

Bobcat 1.3 Bobcat 1.2 (same measurement)
Jev Decision Index 0.3.2, Full score (official; 80% the maintainers' private tests) 62.8, #3 tied of 166 (Jev 60.1) not on the board
Sealed final, four tasks (1,609 decisions, opened once) 94.49% 94.75%
Six-task development evaluation (3,188 decisions) 92.95% 92.94%
TypeSafe's published workflow examples (329 questions) 92.10% 90.88%
Wrong answer named inside the state wins (300 questions x 3 attacks) 6.4% 6.4%
Six public decision sets never used in any Bobcat training (1,800 questions), chance-corrected skill 73.3 71.9
Decision Index 0.3 public suite, half of its items (our run of the official kit) 59.18 58.97
LocalLLaMA/typed-decisions test, odd IDs (never trained on), argmax accuracy 74.2% 73.2%
JevBench public items (231), our measurement (v1.6 method reconstructed), unofficial 77.2 77.8
96K-token inputs (240 questions up to 96,161 tokens) 0 failures, 0 refusals 0 failures

Jev figures are never used; Jev was never called. Differences between Bobcat 1.3 and 1.2 on these sets are small and mostly inside their 95% intervals (see Evaluation).

What it does

Primitive You define Bobcat returns
Choice 1 to 255 named candidates, descriptions optional the top name and a probability for every name
Noul optional meanings of true and false P(true)
Score 1 to 10 ordered levels the expected level and a probability per level

The request and reply use the shapes of TypeSafe's published System One HTTP API, so the official typesafe-sdk works against a Bobcat server by changing its base URL. Inputs over the server's limits are refused with HTTP 422 and never truncated.

Quickstart: serve Bobcat 1.3 on one GPU

curl -LsSf https://astral.sh/uv/install.sh | sh && export PATH="$HOME/.local/bin:$PATH"   # uv
git clone https://github.com/foxl-ai/bobcat && cd bobcat
export UV_PYTHON_PREFERENCE=only-managed
uv venv --python 3.12 .venv-serve
uv pip install --no-config --python .venv-serve/bin/python vllm==0.30.0 fastapi uvicorn scipy jinja2 \
  "tokenizers>=0.21" huggingface_hub typesafe-sdk==0.7.1
export PYTHONPATH=$PWD/src PY=.venv-serve/bin/python

# These weights (about 62 GB), then the typed-decision server, FP8 at load, 96K states
$PY -c "from huggingface_hub import snapshot_download as s; s('sanghwa-na/bobcat-1.3', local_dir='bobcat-1.3')"
VLLM_USE_FLASHINFER_SAMPLER=0 $PY -m bobcat.flash_server --engine vllm --model bobcat-1.3 \
  --compiler-model bobcat-1.3/compiler --quantization fp8 \
  --temperature choice=1.007793,noul=0.445449,score=1.023435 \
  --name bobcat-1.3 --release-date 2026-10-11 --max-num-seqs 64 --max-model-len 98368 \
  --schedule all --engine-arg max_num_batched_tokens=16384 --engine-arg attention_backend=TRITON_ATTN \
  --host 127.0.0.1 --port 8000 --local

Then call it with the official SDK as on the Bobcat Flash 1.2 card, with TYPESAFE_DEFAULT_MODEL="bobcat-1.3".

  • --temperature choice=1.007793,noul=0.445449,score=1.023435 applies one temperature per question type: Choice and Score fitted by log-loss on the even half of the public LocalLLaMA/typed-decisions test set (never trained on), Noul on a held-out calibration split of JevBench-format decisions. A temperature never changes an argmax. --temperature 1.2227 applies the single temperature fitted on the held-out calibration split of the product tasks.
  • --quantization fp8 is vLLM's online FP8 of these BF16 weights (about 31 GB on the GPU). A pre-quantized FP8 checkpoint is at sanghwa-na/bobcat-1.3-fp8. On an 80 GB GPU add --engine-arg kv_cache_dtype=fp8.
  • --max-model-len 98368 reads states up to about 96K tokens in full; longer inputs are refused with HTTP 422. --local disables the shared secret; use it only on a loopback interface.

Running on AWS

One RTX PRO 6000 Blackwell 96 GB (g7e.4xlarge) or one H100 80 GB (p5.4xlarge, FP8 KV cache) runs the Quickstart; the evaluation servers ran one per GPU on B200 hosts (p6-b200.48xlarge). Use a Deep Learning AMI with a recent NVIDIA driver and about 140 GB of disk.

How it was made

  • Three LoRA arms, averaged. Each arm is a rank-16 LoRA (alpha 32) on the base, trained for one epoch (lr 1e-4, 32,768 tokens per GPU on eight GPUs) with the Bobcat 1.2 recipe: KL to a Bobcat teacher where one exists, the gold loss, Score RPS, and KL to the frozen base on every question except the injection pairs. The arms differ only in how many new skill questions they saw (27,000 or 54,000) and the base-KL weight (1.0 or 0.5). Bobcat 1.3 is their uniform average in weight space (each arm merged with bobcat.student_merge, then a float32 weighted sum, BF16 storage). The average was chosen over each single arm by a rule written before it was measured: the development Decision Index half plus six public decision sets that no Bobcat model ever trained on.
  • Data. Bobcat 1.2's training rows without the rows from ARC and BANKING77 (benchmarks of the Decision Index) or overlapping its items (54,732 questions), plus up to 54,000 new skill questions from 17 public sources with commercial-use licences and human-written text and labels: MASSIVE (tool choice, call review and when to call, built from human intents and slots), CommonsenseQA, GAP, QASC, MedMCQA, MedQA, ScienceQA, AQuA-RAT, WiCE, VitaminC, FaithDial, IBM argument stance, a sarcasm headline set, MAUD, LEDGAR, UNFAIR-ToS and ColBERT humor. Datasets whose text was written by a model were left out. Every new question was checked against the Decision Index 0.3 public suite (140,620 requests), the JevBench public items and our evaluation sets (exact strings and word 8-grams); 765 were removed.
  • Labels: the source datasets' own labels only for the new questions; the Bobcat 1.2 rows keep their gold and the Bobcat 1.1 teacher distributions (our earlier model). No GLM, gpt-oss, closed-API or Jev output was used, and no JevBench or Decision Index item was used for training, calibration or prompts.

Evaluation

Differences are paired, with 95% bootstrap intervals over source components. The development and sealed-final figures use the evaluation path (these merged BF16 weights, bobcat.flash_train --eval-only); TypeSafe, injection, long-state, JevBench-format and Decision Index figures use the weights served in FP8 over HTTP.

Sealed final

A sealed split of the product evaluation (1,609 decisions, four tasks, Korean), opened once, after the release manifest that binds these weights, their temperatures and the analysis was frozen. Failures count as wrong (there were none).

Task (decisions) Bobcat 1.3 Bobcat 1.2 Bobcat 1.1 Bobcat Flash 1.1
Search passage selection (500) 82.2% 84.0% 81.0% 77.2%
Citation verification (100) 98.0% 98.0% 100.0% 99.0%
External document screening (550) 98.2% 97.5% 97.5% 96.7%
Request routing (459) 99.6% 99.6% 99.8% 99.6%
Task macro 94.49% 94.75% 94.56% 93.12%
  • Paired: Bobcat 1.3 minus Bobcat 1.2 -0.27 [-0.75, +0.18]; minus Bobcat 1.1 -0.07 [-1.37, +1.10]; minus Flash 1.1 +1.36 [-0.05, +2.76]. Without the 50 pre-declared rows whose text also appears in a training file (none added by this release's data): 94.47%.
  • The release bar fixed before the opening was 92.21%; it was met. Log-loss 0.335, ECE 0.015 at the product temperature.

Development and gates

Bobcat 1.3 Bobcat 1.2
Six-task development evaluation (3,188 decisions), task macro 92.95% 92.94%
... search / citation / tool-call review / screening / routing / classification 91.7 / 99.5 / 94.5 / 95.3 / 99.3 / 77.6 91.0 / 99.3 / 93.9 / 95.5 / 99.3 / 78.8
TypeSafe's 20 workflow examples, agreement (329 questions) 92.10% 90.88%
Wrong answer named inside the state wins (300 questions x 3 attacks) 6.4% 6.4%
... on the external-document screening task / on the other five tasks 37.5% / 0.1% 37.6% / 0.3%
Long states, change at 30K / 61K / 92K tokens -2.5 / -5.3 / -7.0 pt -2.4 / -5.7 / -6.3 pt
96K-token inputs (240 questions up to 96,161 tokens) 0 failures 0 failures

Tool-call review was trained for this release (the MASSIVE tool questions and the product task's own rows are different data); in Bobcat 1.2 it was held out.

Decisions outside the product: held-out and public sets

Bobcat 1.3 Bobcat 1.2
Six public decision sets never in any Bobcat training (GoEmotions, StrategyQA, QuaRTz, PAWS, ETHICS justice, SNIPS; 300 questions each), mean chance-corrected skill 73.3 71.9
LocalLLaMA/typed-decisions test, odd IDs (1,000 questions, never trained on), argmax accuracy 74.2% 73.2%
Decision Index 0.3 public suite, one half of its items (79,326 requests), our run of the official kit 59.18 58.97
  • The Decision Index half was used, with the six held-out sets, to choose among the candidates by a rule written before they were measured, so the 59.18 is optimistic for this model. Areas against Bobcat 1.2 on that half: knowledge +0.1, language +0.7, retrieval -1.0, tools +1.5, arts -0.8.
  • Held-out sets per source (Bobcat 1.3 / 1.2): GoEmotions 53.9 / 54.7, StrategyQA 59.3 / 53.3, QuaRTz 88.0 / 87.3, PAWS 63.3 / 62.7, ETHICS justice 77.3 / 75.3, SNIPS 97.8 / 97.8.

Decision Index 0.3.2 (official)

Measured by the Jev Decision Index maintainers, not by us (Decision Index PR #146, 2026-10-12 batch; read from the board's data/v03.json, generated 2026-10-14T09:16Z). The Full score is 0.20 x public + 0.50 x private tests of the same skills on fresh data + 0.30 x private tasks from new domains, each part equated onto one scale. The private sets are not published.

Bobcat 1.3 Jev, same cases
Full score 62.83 60.11
Rank #3 (tied with Perplexity Decider v1.1 62.75 and Darwin-27B-ZTC 62.46); fifth by score of 166 entries #4 (tied)
Public benchmarks (our submitted run, rescored by the maintainers) 59.31 (#19 on the public score alone) 57.96
Private tests, same skills: raw / on the public scale 59.90 / 59.54 58.04 / 57.66
Private tasks, new domains: raw / on the public scale 58.71 / 70.65 54.98 / 65.63
Requests answered 110,201 of 110,201
Single-request latency, median / p95 (one RTX PRO 6000) 64 / 767 ms
  • The maintainers adopted our public 0.3 run (their rescore, 59.31, equals our claim) and ran the private tests on one RTX PRO 6000 with the Quickstart's serving: vLLM's online FP8 of these BF16 weights at load. Decision agreement with our submitted rows was 98.44% on 900 rows, which passes the board's quantized bar (96%) and would not pass its BF16 bar (98.5%).

  • As the board notes, half of the public suite's items were used to choose among the candidates, so the public part (20% of the Full score) includes items used for selection. The private parts were never available to us.

  • Public items (231): our measurement, unofficial; not a JevBench rank. Three fresh servers as in the Quickstart through the official harness at commit bb05a33, run once after the manifest was frozen: Capability 77.2 (three runs 76.7 / 76.5 / 78.3; Intelligence 69.0, Calibration 85.3). Bobcat 1.2 measured the same way: 77.8. Paired: minus the untrained base +1.3 [-7.2, +8.5]; minus Quyet-1.0-Large (its own runtime, our measurement) +1.6 [-1.3, +7.9].

  • Our own JevBench-format development sets (no JevBench item): Capability 78.7 (Bobcat 1.2 79.3), hardest tier 73.2 (73.7), judgement questions 76.4 (75.7), 61K-96K-token questions 57.7 (48.8).

Latency and throughput

One B200, vLLM 0.30.0, --max-model-len 98368, engine (no HTTP) These weights, FP8 at load FP8 checkpoint
One decision (512 tokens, 8 candidates), p50 28.6 ms 28.9 ms
Independent 1K-token requests, 512 concurrent 21.5K tokens/s 21.5K tokens/s
The longest JevBench-format development item (96K tokens), over HTTP answered, 35.9 s answered, 35.8 s
GPU memory: weights / KV cache 30.6 / 123.3 GiB 30.6 / 123.1 GiB

These were measured on one B200; the Bobcat 1.2 card's H100 80 GB and RTX PRO 6000 figures are for the same architecture and serving command.

Limitations and risks

  • The gains over Bobcat 1.2 are small; most single comparisons are inside their 95% intervals.
  • The Decision Index half-suite figure (59.18) is our run of the public kit on its public items, and those items were used to choose among candidates, so it is optimistic. The official Full score (62.8) adds the maintainers' private sets; its public part includes the same selection items.
  • Tool-call review is now trained (it was held out in 1.2); the transfer check moved to the typed-decisions set.
  • Wrong answers planted in a document under review still win often on the external-document screening task; long states cost accuracy beyond the trained 32,768 tokens.
  • A typed answer can be the wrong valid option, and confidence is a concentration statistic, not the probability of being right.

License and attribution

These weights are released under Apache-2.0 as a derivative of Gemma 4 31B-it by Google DeepMind (Apache License 2.0); LICENSE and NOTICE are included. Training data keep their own licenses (see the code repository's THIRD_PARTY.md). Bobcat is an independent project, not affiliated with or endorsed by TypeSafe AI, Benchmark Heaven, the Decision Index maintainers, Google or any other company named here. The weights are provided as is, without warranty.

Citation

@misc{bobcat13,
  title        = {Bobcat 1.3},
  author       = {{The Bobcat Authors}},
  year         = {2026},
  howpublished = {\url{https://huggingface.co/sanghwa-na/bobcat-1.3}}
}
Downloads last month
335
Safetensors
Model size
31B params
Tensor type
BF16
·
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for sanghwa-na/bobcat-1.3

Finetuned
(387)
this model
Quantizations
2 models

Datasets used to train sanghwa-na/bobcat-1.3

Space using sanghwa-na/bobcat-1.3 1