kairos-door-rain

Binary image classifier: rain vs no_rain, for a camera looking out a dock door. Fine-tuned from facebook/convnext-tiny-224 (Apache-2.0). Part of Kairos, a weather plugin for Boston Dynamics Spot fleets (Knight Hacks IX hackathon build). It feeds one input of a GO / HOLD rule.

Read this first

• Not yet tested on a Spot door view. No such labeled data exists publicly. Every number below is on public photos.
• It does not transfer across sources. In-source calibration (deduplicated): 95.7% rain recall, 1.3% false alarms. On dashcam road photos, AUROC is 0.562: close to a coin flip. Afternoon rain recall is 3.9% (4/102).
• The dashcam numbers are exploratory. That set was seen during development and shaped our data choices, so it is not a clean held-out test.
• Caution-only. In Kairos the camera can add a HOLD but can never turn a weather HOLD into GO. NWS / Open-Meteo rain alone holds the robot. Unknown camera state also holds.

How to use

import { pipeline } from "@huggingface/transformers";
const clf = await pipeline("image-classification", "builtbyd3v/kairos-door-rain", { dtype: "q8" }); // or "fp32"
await clf("door.jpg"); // [{ label: "rain", score }, { label: "no_rain", score }]

Python: pipeline("image-classification", model="builtbyd3v/kairos-door-rain"). Hold when p(rain) >= 0.4447 for q8, or >= 0.4728 for fp32 (each picked on the calibration split, one image per call). Both are in eval_results.json under thresholds.

Results

95% CIs are Wilson score intervals. Threshold rule: the highest threshold that keeps calibration rain recall >= 95%, rounded down. Scored with onnxruntime CPU on the shipped ONNX files, one image per call (as in the browser).

Calibration split (in-source, never trained on, duplicates grouped so none cross splits): 1,033 images (116 rain).

Weights	Threshold	Rain recall	False alarms
fp32	0.4728	95.7% (111/116, CI 90.3-98.2)	1.2% (11/917, CI 0.7-2.1)
q8 (shipped)	0.4447	95.7% (111/116, CI 90.3-98.2)	1.3% (12/917, CI 0.8-2.3)

Weather-Time road images, exploratory (seen during development): 2,600 labeled dashcam photos (595 rainy; 2,005 sunny or cloudy). Never trained on or used for a threshold.

Model	AUROC	Rain recall	False alarm rate	Accuracy
Always "no_rain"	0.5	0.0% (0/595)	0.0% (0/2,005)	77.1%
Baseline dima806/weather_types_image_detection, p(rain) >= 0.5	0.567	53.4% (318/595, CI 49.4-57.4)	41.9% (840/2,005, CI 39.8-44.1)	57.0%
This model fp32, t=0.4728	0.565	43.2% (257/595, CI 39.3-47.2)	32.8% (657/2,005, CI 30.8-34.8)	61.7%
This model q8, t=0.4447 (shipped)	0.562	46.6% (277/595, CI 42.6-50.6)	37.7% (755/2,005, CI 35.6-39.8)	58.7%

Per period, q8, exploratory:

Period	n (rain / no_rain)	Rain recall	False alarms
Morning	1,613 (449 / 1,164)	55.7% (250/449, CI 51.1-60.2)	33.3% (388/1,164, CI 30.7-36.1)
Afternoon	829 (102 / 727)	3.9% (4/102, CI 1.5-9.7)	45.7% (332/727, CI 42.1-49.3)
Dusk	124 (44 / 80)	52.3% (23/44, CI 37.9-66.2)	8.8% (7/80, CI 4.3-17.0)
Dawn	34 (0 / 34)	no rain images	82.3% (28/34, CI 66.5-91.6)

Why it fails, our reading: the training photos are scenic weather shots; the dashcam photos are through a windshield, where rain shows as wiper streaks and wet asphalt. A door camera is a third domain. The next step is labeled frames from the real door camera.

Changes from v2

A review found three evaluation flaws. All fixed here, then retrained:

• Duplicate photos crossed train and calibration (85 SHA-identical groups). prepare.py now groups images by 8x8 average hash before the split: 503 groups (1,439 files) merged. v2's 0.4% calibration false alarm rate was optimistic.
• q8 was scored in batches of 32, but deployment scores one image. Dynamic int8 quantization depends on the batch, so v2's q8 threshold moved from 0.7861 to 0.7666 when re-scored at batch 1. Now batch 1 everywhere.
• One threshold was used for both dtypes. Now there is one per dtype.

Training data and licenses

Source	License	Used as	Images
Xiao, H. (2021). Weather phenomenon database (WEAPD), Harvard Dataverse, doi:10.7910/DVN/M8JQCR, via Kaggle jehanbhathena/weather-dataset	CC0 1.0 (Kaggle and Dataverse)	rain = rain; no_rain = dew, fog/smog, frost, glaze, rainbow, rime, sandstorm, snow. Hail and lightning skipped (ambiguous).	5,894
Ajayi, G. (2018). Multi-class Weather Dataset for Image Classification. Mendeley Data V1, doi:10.17632/4drtyfjtfy.1, via Kaggle pratik2901/multiclass-weather-dataset	CC BY 4.0	rain = Rain; no_rain = Cloudy, Shine, Sunrise	1,125
Kaggle wjybuqi/weathertime-classification-with-road-images (from the CCF BDCI contest)	CC0 1.0 on Kaggle (upstream contest terms not checked)	Exploratory evaluation only. rain = Rainy; no_rain = Cloudy, Sunny. Its unlabeled test folder is unused.	2,600

Split of the two training sources: grouped by 8x8 average hash, then 70 / 15 / 15 over groups (train / val / calibration), stratified by source class, seed 0. Val picks the best epoch; calibration picks the thresholds. Train: 4,946 (513 rain).

Training

training/train.py (reused from spot-wetfloor-convnext-tiny; only labels changed): 4 epochs (best: 3), lr 5e-5, batch 32, light crop/flip/color jitter. Apple M4 Pro (MPS), about 14 minutes. No paid compute. training/export_onnx.py: opset 17, max logit diff vs PyTorch 3e-6. training/quantize.py: dynamic uint8, MatMul/Gemm only (33.8 MB vs 111.4 MB). training/evaluate.py: everything in the tables. Raw numbers: eval_results.json.

Known limits

• Domain gap (above). Treat p(rain) as a weak hint until it is scored on real door frames.
• 8x8 average hash catches exact copies and light re-encodes, not crops or rotations.
• The baseline was trained on the same Kaggle source as ours, so only its dashcam numbers are a fair comparison.
• Night, glare, a dirty lens, and drizzle are untested.