BATCH_INFERENCE.md

Griot Edge batch inference

A standalone inference engine for the Hugging Face Griot Edge package. It runs as a Python library, a one-shot CLI, or an optional persistent HTTP server. The CLI does not start a server. No training repository installation is needed.

Setup

Use Python 3.12 and the dependencies from the published model:

uvx --from huggingface-hub hf download Qlerqly/griot-edge --local-dir model
uv venv --python 3.12 .venv
uv pip install --python .venv/bin/python -r model/requirements.txt

Keep batch_inference.py and batch_server.py together beside this README. CPU, CUDA, and Apple MPS are supported. CUDA selects BF16 automatically when supported; modern MPS selects BF16 when available and CPU selects FP32. Explicit --dtype bf16 never silently falls back. CUDA was benchmarked on an NVIDIA L4 and MPS on an Apple M5 Pro; see the benchmark report for the measured configuration. Measured timings and limitations are in BENCHMARK_REPORT.md. CUDA BF16 showed batch-dependent numerical and transcription differences in our L4 checks; assess recognition accuracy on representative audio before releasing a CUDA service. GPU BF16 remains explicit in the benchmark commands.

One-shot CLI

.venv/bin/python batch_inference.py --model-dir model \
  --audio recording-a.wav recording-b.flac --batch-size 8 --output results.json

Files are downmixed and resampled to 16 kHz. Long recordings use approximately 30-second acoustic windows with 5 seconds of overlap. The centered STFT's endpoint frame is retained for recordings exactly 30 seconds long without creating an unnecessary second chunk. Chunks from all input recordings enter the same scheduler; a long recording can fill a batch on its own.

Each chunk is tagged with its recording, time offset, and retained CTC frame range. Batches are sorted by length, bounded by padded feature frames, and limited to 15% padding by default. Outputs return in original input order. Overlapping frame scores are trimmed and joined before decoding each complete recording. Text strings are not concatenated at chunk boundaries.

Greedy decoding avoids loading KenLM:

.venv/bin/python batch_inference.py --model-dir model --device cpu --greedy \
  --audio recording.wav --batch-size 2 --cpu-threads 6

Alternatively, --manifest inputs.json accepts:

[{"id": "meeting", "audio": "/absolute/path/meeting.wav"},
 {"id": "interview", "audio": "/absolute/path/interview.wav"}]

Controls include --max-batch-frames, --max-padding-fraction, --chunk-seconds, --overlap-seconds, --lookahead-frames, and --decode-workers. Decoder workers are threads sharing one loaded LM; multiple acoustic inference calls on the same engine are serialized.

Python library

import soundfile as sf
from batch_inference import AudioInput, BatchEngine

engine = BatchEngine("model", device="auto", batch_size=8)
audio, sample_rate = sf.read("recording.wav", dtype="float32")
result = engine.transcribe([AudioInput("recording", audio, sample_rate)])
print(result["results"][0]["text"])

Construct the engine once and reuse it to avoid repeated model/LM loading.

Optional persistent server

.venv/bin/python batch_server.py --model-dir model --device auto \
  --dtype bf16 --batch-size 8 --batch-wait-ms 20 --port 8000

The server binds to 127.0.0.1 by default and keeps one acoustic model and LM resident. GET /health reports readiness. POST /transcribe accepts a JSON batch of WAV/FLAC (or other libsndfile-supported) files encoded as base64:

import base64
import json
from pathlib import Path
from urllib.request import Request, urlopen

payload = {
    "inputs": [
        {"id": name, "audio_base64": base64.b64encode(Path(name).read_bytes()).decode()}
        for name in ["recording-a.wav", "recording-b.flac"]
    ]
}
request = Request(
    "http://127.0.0.1:8000/transcribe",
    data=json.dumps(payload).encode(),
    headers={"Content-Type": "application/json"},
)
with urlopen(request) as response:
    print(json.load(response))

Queued requests are collected for up to 20 ms and their chunks share GPU batches. Request IDs remain independent even when clients choose identical input IDs. The queue defaults to 32 waiting requests and each group to eight requests. Queue overflow returns 503; invalid input returns 400; oversized bodies return 413; timed-out requests return 504. Per-request limits are 16 inputs, 64 MiB JSON, 256 MiB decoded float32 audio, and one hour of audio. Run one server process per device. This is a local reference server; public deployment needs a suitable HTTP frontend and authentication supplied by the hosting environment.

stats describes the entire coalesced group; request_seconds, queue_seconds, and group_requests describe the individual request.

SIGTERM and SIGINT stop acceptance of new jobs and shut the server down. Container and native deployment commands are in DEPLOYMENT.md.

Benchmarks and verification

.venv/bin/python test_batch.py
.venv/bin/python benchmark_batch.py --model-dir model --sample sample.wav \
  --device mps --dtype bf16 --hardware-label "Apple M5 Pro GPU" --output benchmark.json

Benchmark workloads are eight 30-second inputs, mixed-length inputs, and a single four-minute recording, using repeated synthetic English speech. They measure capacity, not speech accuracy. Each configuration has a warmup and three timed runs. Feature extraction and decoding are included in total time; model startup, file reading, and HTTP transfer are excluded and reported or tested separately. Audio, features, and joined CTC scores remain in CPU memory; only acoustic model windows/batches are bounded on the device.

The large GPU capacity sweep uses 128 thirty-second inputs, 100 mixed-duration inputs, and one hour-long recording. It uses greedy decoding to expose GPU scaling independently of CPU KenLM cost; the smaller default suite measures full beam-decoding throughput. Do not compare the two modes as if they were the same end-to-end workload.

.venv/bin/python benchmark_batch.py --model-dir model --sample sample.wav \
  --device mps --dtype bf16 --hardware-label "Apple M5 Pro GPU" \
  --batch-sizes 1 2 4 8 16 32 64 128 --large-workloads --greedy \
  --output benchmark-large.json

Effective batch sizes, padding, and memory are included in the JSON. An OOM is reported as an OOM, without reducing the requested batch size behind the scenes. With three repeats, p95 is descriptive of those runs, not a production tail latency estimate. Coalesced HTTP requests finish after their group completes; a long recording can delay shorter requests in the same group.

For an HTTP request containing 128 full 30-second WAV inputs, increase the request limits explicitly, independently of GPU batch size:

.venv/bin/python batch_server.py --model-dir model --device mps --dtype bf16 \
  --batch-size 128 --max-inputs 128 --max-body-mib 256 --max-audio-seconds 7200

For a one-time L4 comparison, install the Modal client, authenticate it, and run python -m modal run modal_benchmark.py --large. This creates an ephemeral benchmark function, not a persistent hosted server. python -m modal run modal_validate.py reproduces the L4 BF16/FP32 numerical diagnostic; FP32 here is a correctness reference, not a GPU performance result.

Decoder selection and tuning

The native CLI/library/server default to pyctcdecode. Install requirements-flashlight.txt to enable the optional backend:

uv pip install --python .venv/bin/python -r requirements-flashlight.txt
.venv/bin/python batch_inference.py --model-dir model --device mps --dtype bf16 \
  --audio recording.wav --decoder-backend flashlight --flashlight-token-beam-size 8

Flashlight token beam 8 is the fast tested setting; 16 is available for quality comparison and 151 removes token-count pruning. Beam width defaults to 25. --beam-prune-logp and --token-min-logp tune pyctcdecode; native search uses --flashlight-beam-threshold. --lm-alpha, --lm-beta, --kenlm, and --unigrams configure fusion or a custom LM. The server accepts all these options, and the original inference.py supports file/live decoder selection.

The Docker image bundles both decoders and defaults to the fast Flashlight settings. All command/environment overrides and server/CLI modes are documented in DEPLOYMENT.md. GPU commands use BF16, CPU uses FP32. Flashlight's word lexicon changes search behavior and constrains unseen words; see FLASHLIGHT_REPORT.md before changing quality-sensitive workflows. Greedy decoding remains available with --greedy.