---
license: apache-2.0
title: distill-pipeline
tags:
  - distillation
  - retrieval-augmented-generation
  - rag
  - synthetic-data
  - pipeline
  - nodejs
  - ollama
  - self-improvement
  - question-generation
  - reward-modeling
---
# distill-pipeline — modular synthetic data engine (thinking + instruct)

`distill-pipeline` is a modular **Node.js synthetic data pipeline** that reads JSONL inputs, runs generation + verification + reward, and writes JSONL outputs. It supports both:

- **“Thinking” generators** that produce visible reasoning, and
- **“Instruct” generators** that produce direct answers,

with **separate caches and outputs** so you can compare styles without mixing artefacts.

Rather than owning retrieval, `distill-pipeline` is designed as the **middle layer** in a stack: you feed it JSONL chunks or questions (for example, from [`distill-rag`](https://huggingface.co/htaf/distill-rag) or your own tooling), and it orchestrates the LLM stages to produce clean, reusable synthetic data.

---

## What it does

- **JSONL-first pipeline**
  - Reads JSONL chunks (default `data/rag_chunks.jsonl`) or static question seeds (`test_samples/seed_questions.jsonl`).
  - Writes accepted samples as JSONL into `gold/*.jsonl`.

- **Two pipeline modes**
  - **Thinking mode:** question → reasoning-style answer → verification → reward.
  - **Instruct mode:** instruction → direct answer pairs, for fine-tuning assistants.
  - Each mode has its own cache + output paths so you can run them independently.

- **Retrieval-agnostic, RAG-friendly**
  - Works with plain JSONL; any RAG stack or pre-processing step that can emit JSONL chunks can plug in.
  - Optional “question-first” mode uses context chunks (or Elasticsearch) to generate questions from your corpus.

- **Stage-based and cache-heavy**
  - Questions, generations, verifications, and rewards are **cached on disk** (JSONL).
  - You can change prompts or models and reuse existing work instead of re-running everything.

- **Local-first providers**
  - Built to run locally with **Ollama** as the default provider for all stages.
  - Also supports OpenAI/HTTP-style providers, plus mock providers for tests/benchmarks.

- **Monitoring and benchmarks**
  - Live HUD (`scripts/live_bench.mjs`) for real-time throughput/accept-rate monitoring.
  - Benchmark script (`scripts/bench_pipeline.mjs`) to measure pipeline speed without burning GPU on real models.

---

## Quickstart

```bash
git clone https://github.com/elspru/distill-pipeline && cd distill-pipeline
npm install

# Generate reasoning data to aid in complex problem-solving models
PIPELINE_SEED_MODE=question-first PIPELINE_RANDOM_WALK=1 npm run pipeline -- --limit 10 --verbose

# Create instruction data for fine-tuning helpful assistants
INSTRUCT_PIPELINE=1 INSTRUCT_GENERATOR_MODEL=phi-4-instruct npm run pipeline -- --out gold/instruct_gold.jsonl --verbose
```

For ongoing generation to build larger datasets:

```bash
scripts/run_thinking_continuous.sh
INSTRUCT_GENERATOR_MODEL=phi-4-instruct scripts/run_instruct_continuous.sh
```

## Configuration (see `.env.example`)

```
# Retrieval
ES_NODE=http://localhost:9200
ES_INDEX=quo_distill_index
EMBED_URL=http://localhost:11434/api/embeddings
EMBED_MODEL=mxbai-embed-large
# Providers per stage
GENERATOR_PROVIDER=ollama
VERIFIER_PROVIDER=ollama
REWARD_PROVIDER=ollama
QUESTION_PROVIDER=ollama
# Models
GENERATOR_MODEL=qwen3-vl:8b-thinking
VERIFIER_MODEL=tensortemplar/patronus-lynx:8b-instruct-q4_K_M
REWARD_MODEL=tensortemplar/patronus-lynx:8b-instruct-q4_K_M
QUESTION_MODEL=qwen2.5-7b-instruct
# Instruct-only generator
INSTRUCT_PIPELINE=0
INSTRUCT_GENERATOR_MODEL=phi-4-instruct
INSTRUCT_GENERATOR_PROVIDER=ollama
# Pipeline knobs
PIPELINE_SEED_MODE=question-first
PIPELINE_RANDOM_WALK=0 # set 1 for shuffled chunks
QUESTION_MAX_PER_CHUNK=5
# PIPELINE_CHUNK_LIMIT=10
# PIPELINE_CACHE_DIR=data/cache # override (e.g., data/cache_instruct)
```

## Key scripts
- `npm run pipeline` — main pipeline CLI (`--limit`, `--out`, `--chunk-limit`, `--verbose`).
- `scripts/run_thinking_continuous.sh` — loop thinking pipeline with random walk.
- `scripts/run_instruct_continuous.sh` — loop instruct pipeline (needs `INSTRUCT_GENERATOR_MODEL`).
- `scripts/try_generator_prompt.sh` — send generator prompt with cached chunk/question (`--random`, `-r` for reasoning).
- `scripts/cache_report.mjs` — cache stats; set `CACHE_REPORT_MODE=thinking|instruct|both` or `PIPELINE_CACHE_DIR=...`.
- `scripts/bench_pipeline.mjs` — mock-provider throughput benchmark (question-first); use `--limit`, `--chunk-limit`, `--random-walk`, `--cache-dir`.
- `scripts/live_bench.mjs` — live HUD (readline) showing throughput/accept rate/status; defaults to mock providers; `--real` to use real providers.

## Outputs
- Gold JSONL default: `gold/pipeline_gold.jsonl` (instruct default: `gold/pipeline_gold_instruct.jsonl`).
- Sample gold: `samples/pipeline_gold_sample.jsonl`.
- Cache defaults: `data/cache` (thinking) and `data/cache_instruct` (instruct); both gitignored.

## Hugging Face / GitHub distribution
- License: Apache-2.0 (`LICENSE`).
- CI: `.github/workflows/ci.yml` runs `npm test` on push/PR.
- Push to GitHub:
  ```
  git remote add origin https://github.com/elspru/distill-pipeline
  git push origin main
  ```
- Push to Hugging Face (user: htaf):
  ```
  git lfs install
  git remote add hf https://huggingface.co/htaf/distill-pipeline
  git push origin main
  git push hf main
  ```
- Publish code + prompts + `samples/pipeline_gold_sample.jsonl`. Keep caches/gold outputs out (gitignored).

## Project structure
```
prompts/ # stage prompts
src/ # pipeline, providers, stages
tests/ # Vitest
data/ # rag chunks (jsonl), cache (ignored)
gold/ # outputs (ignored)
scripts/ # tooling + runners
samples/pipeline_gold_sample.jsonl
```

## Testing
`npm test`

## License
Apache-2.0

Join the movement: Use `distill-pipeline` to create and share datasets that elevate AI for everyone.