jatshi/StreamSense-Serve-v4-Router

StreamSense-Serve 4.0 Router

This repository contains the trained, auditable routing artifact used by StreamSense-Serve 4.0. It is not a copy of the Qwen or BGE base-model weights. The artifact stores four lightweight decision branches and their jointly calibrated thresholds:

1. safety risk;
2. lightweight-evidence uncertainty;
3. cross-modal conflict;
4. visual dependency.

The runtime escalates to visual enhancement when any branch crosses its threshold. Risk and visual dependency use frozen BAAI/bge-small-zh-v1.5 embeddings followed by logistic heads; uncertainty and conflict use physical signal and cross-modal features. The BGE revision is pinned inside router_artifact.json and must be downloaded under its upstream terms.

Files

• router_artifact.json: coefficients, normalizers, frozen encoder revision, four thresholds, training-data digest, and grouped-OOF metrics;
• router_test_metrics.json: results on 576 physical media cases from eight unseen semantic groups, including baselines, condition metrics, robustness slices, and group bootstrap;
• e2e_summary.json: formal 32-video × three-system result, including answer, citation, visual-call, and total-backend-call rates;
• e2e_diagnostics.json: group-bootstrap confidence intervals, paired system differences, and failure-mode counts;
• SHA256SUMS: integrity hashes for the published artifacts;
• MODEL_CARD.md: this card in a standalone filename for local archives.

Measured result

Metric	Value
Development semantic groups	24
Grouped OOF folds	4
OOF escalation recall	95.88%
Held-out semantic groups	8
Held-out physical cases	576
Held-out escalation recall	94.81%
Held-out risk recall	98.61%
Visual-enhancement rate	94.97%

The held-out split is deliberately difficult: 520/576 cases are oracle-positive, so even a perfect router could avoid visual enhancement on at most 9.72% of this split. The measured 5.03% reduction is not a production-traffic cost estimate.

The separate formal end-to-end run used 32 real MP4 cases sampled evenly from eight untouched semantic groups and executed learned, always-visual, and never-visual systems once each (96 system records). Quality pass rates were 65.63%, 68.75%, and 78.13%, respectively. The preregistered H5 hypothesis therefore failed: in this test, OCR already resolved the two visual groups and the free-text VLM evidence introduced state inconsistencies instead of net quality. This negative result is retained because it defines the next work item: non-OCR visual tasks and structured VLM evidence. Latency values in this run are not comparable across systems because ASR/OCR outputs were memoized in execution order.

Reproduce

python -m pip install -e ".[semantic-router]"
python scripts/evaluate_v4_router.py \
  --features /path/to/features.jsonl \
  --router router_artifact.json \
  --output reproduced_metrics.json \
  --bootstrap-iterations 1000

The complete media builder, Faster-Whisper/RapidOCR extraction, group-isolated OOF training, and end-to-end Qwen-VL/EvidenceAgent evaluation are in the GitHub repository. See the implementation status, release notes and failure diary, and learning guide.

Intended use and limitations

This artifact is an engineering benchmark router for evidence-grounded audio/video assistants. It was trained on 32 generated Chinese semantic scenarios expanded into 2,304 physical corruption conditions. It is not validated for medical decisions, surveillance, identity recognition, or autonomous safety actions. The corpus uses one synthetic voice family and controlled visual templates; accents, spontaneous speech, natural meetings, and deployment priors require separate evaluation.

The endpoint experiment uses one text EvidenceAgent call per case and adds one image-bearing VLM call only when the router escalates. Therefore visual-enhancement rate does not mean that all non-escalated cases avoid every large-model call.