griot-edge / BATCH_INFERENCE.md

Add WebSocket live streaming, concurrent HTTP batch serving, and silence-aware >30s VAD chunking

558285e verified 3 days ago

6.34 kB

# Griot Edge batch inference

A standalone inference engine for the Hugging Face Griot Edge package. It runs
as a Python library, a one-shot CLI (`batch_inference.py`), a lightweight stdlib
HTTP server (`batch_server.py`), or behind the full FastAPI WebSocket + HTTP
serving stack (`serve.py`, documented in [SERVING.md](SERVING.md)).
The CLI does not start a server. No training repository installation is needed.

## Setup

Use Python 3.12 and the dependencies from the published model:

```bash
uvx --from huggingface-hub hf download Qlerqly/griot-edge --local-dir model
uv venv --python 3.12 .venv
uv pip install --python .venv/bin/python -r model/requirements.txt
```

Keep `batch_inference.py`, `chunking.py`, `vad.py`, `ctc_alignment.py`, and
`batch_server.py` together beside `inference.py` and `runtime.py`.
CPU, CUDA, and Apple MPS are supported. CUDA selects BF16 automatically when
supported; modern MPS selects BF16 when available and CPU selects FP32.
Explicit `--dtype bf16` never silently falls back. CUDA was benchmarked on an NVIDIA L4 and
MPS on an Apple M5 Pro; see [BENCHMARK_REPORT.md](BENCHMARK_REPORT.md) for the measured configuration.
CUDA BF16 showed batch-dependent numerical and transcription differences in
our L4 checks; assess recognition accuracy on representative audio before
releasing a CUDA service. GPU BF16 remains explicit in the benchmark commands.

## One-shot CLI

```bash
.venv/bin/python batch_inference.py --model-dir model \
  --audio recording-a.wav recording-b.flac --batch-size 8 --output results.json
```

Files are downmixed and resampled to 16 kHz. By default (`--chunk-mode vad`),
recordings longer than 30 seconds are split at natural speech pauses between
20.0 s and 28.0 s using vectorized energy VAD (`vad.py`). Clean pause cuts use
0.0 s overlap, while uninterrupted speech falls back to 28.0 s windows with
1.5 s overlap stitched via CTC Viterbi word timestamps (`ctc_alignment.py`) and
Needleman-Wunsch dynamic-programming alignment (`chunking.py`) so words are
neither clipped nor duplicated at boundaries.

Pass `--chunk-mode fixed` to use fixed 30-second acoustic windows with 5 seconds
of overlap, where overlapping CTC frame scores are trimmed at their global
midpoint before a single decode per recording. Chunks from all input recordings
enter the same scheduler; a long recording can fill a batch on its own, and
outputs always return in original input order.

Greedy decoding avoids loading KenLM:

```bash
.venv/bin/python batch_inference.py --model-dir model --device cpu --greedy \
  --audio recording.wav --batch-size 2 --cpu-threads 6
```

Alternatively, `--manifest inputs.json` accepts:

```json
[{"id": "meeting", "audio": "/absolute/path/meeting.wav"},
 {"id": "interview", "audio": "/absolute/path/interview.wav"}]
```

Controls include `--chunk-mode {vad,fixed}`, `--min-chunk-seconds`,
`--max-chunk-seconds`, `--vad-threshold`, `--vad-min-silence-seconds`,
`--max-batch-frames`, `--max-padding-fraction`, `--chunk-seconds`,
`--overlap-seconds`, `--lookahead-frames`, and `--decode-workers`. Decoder
workers are threads sharing one loaded LM; multiple acoustic inference calls on
the same engine are serialized.

## Python library

```python
import soundfile as sf
from batch_inference import AudioInput, BatchEngine
engine = BatchEngine("model", device="auto", batch_size=8, chunk_mode="vad")
audio, sample_rate = sf.read("recording.wav", dtype="float32")
result = engine.transcribe([AudioInput("recording", audio, sample_rate)])
print(result["results"][0]["text"])
```

Construct the engine once and reuse it to avoid repeated model/LM loading.

## Persistent servers (`serve.py` vs `batch_server.py`)
For production WebSocket live streaming (`WS /v1/stream`) alongside concurrent
HTTP single and batch endpoints (`POST /v1/transcribe`, `POST /v1/transcribe/batch`),
use `serve.py` (requires `requirements-server.txt`; see [SERVING.md](SERVING.md)):
```bash
uv pip install --python .venv/bin/python -r model/requirements-server.txt
.venv/bin/python serve.py --model-dir model --device auto --dtype auto \
  --max-batch-size 8 --batch-wait-ms 20 --port 8000
```
For zero-web-framework environments using only the Python standard library,
`batch_server.py` remains available:

```bash
.venv/bin/python batch_server.py --model-dir model --device auto \
  --dtype bf16 --batch-size 8 --batch-wait-ms 20 --port 8000
```

`batch_server.py` binds to `127.0.0.1` by default, exposes `GET /health` and
`POST /transcribe`, collects queued requests for up to 20 ms (up to 8 requests
per group, queue depth 32), and enforces per-request limits (16 inputs, 64 MiB
JSON, 256 MiB decoded float32 audio, 1 hour audio).

Container and native deployment commands are in [DEPLOYMENT.md](DEPLOYMENT.md).

## Benchmarks and verification

```bash
.venv/bin/python test_batch.py
.venv/bin/python test_chunking_vad.py
.venv/bin/python benchmark_server.py --model-dir model --device mps --dtype bf16
.venv/bin/python benchmark_batch.py --model-dir model --sample sample.wav \
  --device mps --dtype bf16 --hardware-label "Apple M5 Pro GPU" --output benchmark.json
```

See [BENCHMARK_REPORT.md](BENCHMARK_REPORT.md) for live ASGI server streaming
and concurrent HTTP numbers alongside full Apple M5 Pro and NVIDIA L4 batch
sweeps.

## Decoder selection and tuning

The native CLI/library/server default to `pyctcdecode`. Install
`requirements-flashlight.txt` to enable the optional C++ Flashlight backend:

```bash
uv pip install --python .venv/bin/python -r requirements-flashlight.txt
.venv/bin/python batch_inference.py --model-dir model --device mps --dtype bf16 \
  --audio recording.wav --decoder-backend flashlight --flashlight-token-beam-size 8
```

Flashlight token beam 8 is the fast tested setting; 16 is available for quality
comparison and 151 removes token-count pruning. Beam width defaults to 25.
`--beam-prune-logp` and `--token-min-logp` tune `pyctcdecode`; native search uses
`--flashlight-beam-threshold`. `--lm-alpha`, `--lm-beta`, `--kenlm`, and
`--unigrams` configure fusion or a custom LM. Flashlight's word lexicon changes
search behavior and constrains unseen words; see [FLASHLIGHT_REPORT.md](FLASHLIGHT_REPORT.md)
before changing quality-sensitive workflows. Greedy decoding remains available
with `--greedy` (or `--decoder greedy` on `serve.py`).

	# Griot Edge batch inference

	A standalone inference engine for the Hugging Face Griot Edge package. It runs
	as a Python library, a one-shot CLI (`batch_inference.py`), a lightweight stdlib
	HTTP server (`batch_server.py`), or behind the full FastAPI WebSocket + HTTP
	serving stack (`serve.py`, documented in [SERVING.md](SERVING.md)).
	The CLI does not start a server. No training repository installation is needed.

	## Setup

	Use Python 3.12 and the dependencies from the published model:

	```bash
	uvx --from huggingface-hub hf download Qlerqly/griot-edge --local-dir model
	uv venv --python 3.12 .venv
	uv pip install --python .venv/bin/python -r model/requirements.txt
	```

	Keep `batch_inference.py`, `chunking.py`, `vad.py`, `ctc_alignment.py`, and
	`batch_server.py` together beside `inference.py` and `runtime.py`.
	CPU, CUDA, and Apple MPS are supported. CUDA selects BF16 automatically when
	supported; modern MPS selects BF16 when available and CPU selects FP32.
	Explicit `--dtype bf16` never silently falls back. CUDA was benchmarked on an NVIDIA L4 and
	MPS on an Apple M5 Pro; see [BENCHMARK_REPORT.md](BENCHMARK_REPORT.md) for the measured configuration.
	CUDA BF16 showed batch-dependent numerical and transcription differences in
	our L4 checks; assess recognition accuracy on representative audio before
	releasing a CUDA service. GPU BF16 remains explicit in the benchmark commands.

	## One-shot CLI

	```bash
	.venv/bin/python batch_inference.py --model-dir model \
	--audio recording-a.wav recording-b.flac --batch-size 8 --output results.json
	```

	Files are downmixed and resampled to 16 kHz. By default (`--chunk-mode vad`),
	recordings longer than 30 seconds are split at natural speech pauses between
	20.0 s and 28.0 s using vectorized energy VAD (`vad.py`). Clean pause cuts use
	0.0 s overlap, while uninterrupted speech falls back to 28.0 s windows with
	1.5 s overlap stitched via CTC Viterbi word timestamps (`ctc_alignment.py`) and
	Needleman-Wunsch dynamic-programming alignment (`chunking.py`) so words are
	neither clipped nor duplicated at boundaries.

	Pass `--chunk-mode fixed` to use fixed 30-second acoustic windows with 5 seconds
	of overlap, where overlapping CTC frame scores are trimmed at their global
	midpoint before a single decode per recording. Chunks from all input recordings
	enter the same scheduler; a long recording can fill a batch on its own, and
	outputs always return in original input order.

	Greedy decoding avoids loading KenLM:

	```bash
	.venv/bin/python batch_inference.py --model-dir model --device cpu --greedy \
	--audio recording.wav --batch-size 2 --cpu-threads 6
	```

	Alternatively, `--manifest inputs.json` accepts:

	```json
	[{"id": "meeting", "audio": "/absolute/path/meeting.wav"},
	{"id": "interview", "audio": "/absolute/path/interview.wav"}]
	```

	Controls include `--chunk-mode {vad,fixed}`, `--min-chunk-seconds`,
	`--max-chunk-seconds`, `--vad-threshold`, `--vad-min-silence-seconds`,
	`--max-batch-frames`, `--max-padding-fraction`, `--chunk-seconds`,
	`--overlap-seconds`, `--lookahead-frames`, and `--decode-workers`. Decoder
	workers are threads sharing one loaded LM; multiple acoustic inference calls on
	the same engine are serialized.

	## Python library

	```python
	import soundfile as sf
	from batch_inference import AudioInput, BatchEngine

	engine = BatchEngine("model", device="auto", batch_size=8, chunk_mode="vad")
	audio, sample_rate = sf.read("recording.wav", dtype="float32")
	result = engine.transcribe([AudioInput("recording", audio, sample_rate)])
	print(result["results"][0]["text"])
	```

	Construct the engine once and reuse it to avoid repeated model/LM loading.

	## Persistent servers (`serve.py` vs `batch_server.py`)

	For production WebSocket live streaming (`WS /v1/stream`) alongside concurrent
	HTTP single and batch endpoints (`POST /v1/transcribe`, `POST /v1/transcribe/batch`),
	use `serve.py` (requires `requirements-server.txt`; see [SERVING.md](SERVING.md)):

	```bash
	uv pip install --python .venv/bin/python -r model/requirements-server.txt
	.venv/bin/python serve.py --model-dir model --device auto --dtype auto \
	--max-batch-size 8 --batch-wait-ms 20 --port 8000
	```

	For zero-web-framework environments using only the Python standard library,
	`batch_server.py` remains available:

	```bash
	.venv/bin/python batch_server.py --model-dir model --device auto \
	--dtype bf16 --batch-size 8 --batch-wait-ms 20 --port 8000
	```

	`batch_server.py` binds to `127.0.0.1` by default, exposes `GET /health` and
	`POST /transcribe`, collects queued requests for up to 20 ms (up to 8 requests
	per group, queue depth 32), and enforces per-request limits (16 inputs, 64 MiB
	JSON, 256 MiB decoded float32 audio, 1 hour audio).

	Container and native deployment commands are in [DEPLOYMENT.md](DEPLOYMENT.md).

	## Benchmarks and verification

	```bash
	.venv/bin/python test_batch.py
	.venv/bin/python test_chunking_vad.py
	.venv/bin/python benchmark_server.py --model-dir model --device mps --dtype bf16
	.venv/bin/python benchmark_batch.py --model-dir model --sample sample.wav \
	--device mps --dtype bf16 --hardware-label "Apple M5 Pro GPU" --output benchmark.json
	```

	See [BENCHMARK_REPORT.md](BENCHMARK_REPORT.md) for live ASGI server streaming
	and concurrent HTTP numbers alongside full Apple M5 Pro and NVIDIA L4 batch
	sweeps.

	## Decoder selection and tuning

	The native CLI/library/server default to `pyctcdecode`. Install
	`requirements-flashlight.txt` to enable the optional C++ Flashlight backend:

	```bash
	uv pip install --python .venv/bin/python -r requirements-flashlight.txt
	.venv/bin/python batch_inference.py --model-dir model --device mps --dtype bf16 \
	--audio recording.wav --decoder-backend flashlight --flashlight-token-beam-size 8
	```

	Flashlight token beam 8 is the fast tested setting; 16 is available for quality
	comparison and 151 removes token-count pruning. Beam width defaults to 25.
	`--beam-prune-logp` and `--token-min-logp` tune `pyctcdecode`; native search uses
	`--flashlight-beam-threshold`. `--lm-alpha`, `--lm-beta`, `--kenlm`, and
	`--unigrams` configure fusion or a custom LM. Flashlight's word lexicon changes
	search behavior and constrains unseen words; see [FLASHLIGHT_REPORT.md](FLASHLIGHT_REPORT.md)
	before changing quality-sensitive workflows. Greedy decoding remains available
	with `--greedy` (or `--decoder greedy` on `serve.py`).