Hugging Face's logo

markov-ai
/
autocad-bench

benchmark
computer-use
autocad
cad
multimodal
Model card Files
xet
 Community
autocad-bench / CONTRIBUTING.md
Dev Mandal
Initial open-source release
0be8f22
preview code
|
Raw
History Blame Contribute Delete
3.44 kB

Contributing

AutoCAD Bench separates task data, model execution, trusted evaluation, and infrastructure. Keep changes inside the narrowest applicable boundary and preserve the scorer/model trust split.

Development setup 

uv sync --dev
cd apps/dashboard && npm ci

Common changes

Add or update a task

  1. Add the reference PNG and gold DWG under tasks/corpus/2d/ or tasks/corpus/3d/.
  2. Add or update the canonical row in tasks/manifest.jsonl.
  3. Update tasks/gold-audit.jsonl with verified file metadata.
  4. Rebuild the gold cache with the pinned trusted evaluator.
  5. Run corpus, audit, cache, and test validation.

Never expose a gold path through PublicTaskSpec, AutoCADTaskData, a model prompt, or a model trace.

Change the harness

Model-provider logic and the agent loop belong under src/autocad_bench/harness/. Computer actions must remain typed and fully validated before any broker call. Preserve the 7,000-action rollout budget and the distinction between audit-only intermediate frames and model-visible observations.

Change evaluation

Trusted evaluation belongs under src/autocad_bench/evaluation/. A render-contract change requires a new evaluator version and gold cache; never reinterpret an old version in place. Windows evaluator changes also require a compiled fresh-clone gate before an AMI becomes active.

Change infrastructure

Backend selection, registration, and lifecycle boundaries belong under src/autocad_bench/infrastructure/. Keep the batch controller backend-neutral. New external backends should implement the public plugin contract and register an autocad_bench.infrastructure entry point; see docs/infrastructure-plugins.md. Backends that retain reconnectable workers should also implement the optional recovery protocol; do not put provider-specific worker inspection back into orchestration/resume.py.

The built-in AWS lifecycle and broker-client code remains under src/autocad_bench/sandbox/. Windows runtime source belongs under sandbox/windows/; cloud-controller assets belong under infra/cloud-controller/. Do not launch instances, deploy infrastructure, or replace the active AMI without explicit approval.

Validation

Run the complete offline gate before handing off a change:

uv run autocad-bench validate
uv run autocad-bench validate-audit
uv run pytest

cd apps/dashboard
npm test
npm run build

Before publishing, build the package and corpus into one new release directory, then verify that complete directory:

uv run autocad-bench-release build \
  --output dist/releases/0.1.0
uv run autocad-bench-release verify \
  --root dist/releases/0.1.0

The builder uses the allowlist-based bundle exporter and pinned PEP 517 dependencies. It must never be expanded to include configs, environment files, controller state, or run outputs. See docs/releases.md for the complete manual publishing and reproducibility workflow.

When a current evaluator cache is available locally:

uv run autocad-bench verify-gold-cache \
  --evaluator-version autocad-2019-r23.0.46-v4 \
  --split all

Live provider or AWS checks are separate from the offline suite and require explicit credentials.

Documentation

Update the relevant contract whenever behavior changes. Release evidence is append-only: add a new release record instead of editing historical results to describe a newer runtime.