Deterministic Final-DWG Scorer Contract

This scorer is the official reproducible benchmark score. The checksum-bound vision judge described in vision-judge.md remains an independent calibration and audit signal; it is not blended into the deterministic result.

The correctness reward is computed only from the final attempt.dwg. Action traces, screenshots, action count, elapsed time, and whether the model independently saved are diagnostic data and are not scorer inputs.

Inputs

Opaque task_id
Candidate DWG bytes
Active evaluator version
Matching gold-cache store
Versioned scorer configuration

The scorer rejects missing candidates and files that do not have the benchmark's pinned AC1032 DWG signature before calling the trusted evaluator.

Trusted evaluator result

The private evaluator returns:

Validity and a structured failure reason
AutoCAD and evaluator versions
Render-configuration checksum
Candidate source checksum
Entity count and entity-type counts
Prohibited entity types, if any
Model-space extents in drawing units
Task drawing type (2d or 3d), true-3D entity count, and XYZ model extents
Paper-layout entity/annotation inventories and floating viewport camera data
The rendered space/layout and deterministic selection reason
A fixed-size normalized monochrome PNG render
Render checksum
Sampled world-coordinate paths for model-space curves, independent of entity segmentation
Dimension family, measured/displayed value, text position, rotation, and anchor points
Standalone text annotation content, position, height, and rotation

The following failures force a zero reward:

missing_candidate
invalid_dwg_signature
unreadable_dwg
empty_model_space
prohibited_entity
external_reference
render_failed
evaluator_unavailable
evaluator_error
gold_cache_missing
gold_cache_invalid

Deterministic comparison (score schema v3)

The scorer has explicit 2D and 3D paths. It never selects a method based on whether an incidental curve happens to exist in a DWG.

2D vector-semantic comparison (v4)

raw = weighted_mean(applicable components only)
geometry_gate = 0.35 + 0.65 * geometry_similarity
score = min(raw, geometry_gate)

components:
  0.60 * geometry_similarity
  0.25 * dimension_similarity
  0.10 * standalone_text_annotation_similarity
  0.05 * registered_visual_similarity

If the gold contains no dimensions or no standalone text, that component is marked non-applicable and the remaining weights are normalized. It does not silently award a perfect score.

The evaluator samples LINE, polyline, arc, circle, ellipse, spline, and equivalent model-space curves. Python resamples them at a gold-derived physical interval, translates bounding-box centers together, and computes a symmetric soft Chamfer/F1 score. LEADER/MULTILEADER shafts are annotation rather than part geometry. Isolated line paths that cross the object without terminating on another edge are treated as centerlines or center marks and excluded from the base-geometry cloud. Entity segmentation does not matter; rotation, reflection, missing paths, and extra paths are not aligned away. Uniform scale remains significant unless the gold explicitly says that no dimensions are required or that the drawing is not to scale.

Dimensions are matched by semantic family, visibly displayed value, and translated text position. Correct TEXT/MTEXT radius, diameter, angular, and plain numeric callouts are promoted to dimensions, so manual drafting is equivalent to native DIMENSION entities. AutoCAD symbols, decimal commas, and multiplicity prefixes such as 2xR1.4 are normalized. A native text override is authoritative for this component because it is what the submitted sheet shows; the geometry component and gate independently prevent a label from concealing wrong part geometry. Linear tolerance is the larger of 0.05 drawing units or 0.5%; angular tolerance is 0.25 degrees. Measurement correctness contributes 80% and placement 20%.

Standalone TEXT, MTEXT, and text-bearing MULTILEADER annotations are matched separately by normalized content and translated placement. AutoCAD formatting codes, case, and standard diameter/degree encodings are normalized.

Candidate and gold monochrome masks are centroid-registered and compared with an eight-pixel tolerance. This visual component carries only 5% for 2D.

3D layout-spatial comparison (v2)

3D scoring uses the required Layout1 presentation and genuine model-space 3D facts. It never treats model-space 2D curve paths as solid topology.

raw = weighted_mean(applicable components only)
geometry_gate = 0.35 + 0.65 * geometry_similarity
score = min(raw, geometry_gate)

components:
  0.60 * geometry_similarity
  0.25 * Layout1_dimension_similarity
  0.10 * Layout1_text_annotation_similarity
  0.05 * presentation_similarity

geometry_similarity = true_3d_presence
                    * (0.60 * normalized_Layout1_visual_similarity
                     + 0.40 * isolated_XYZ_extent_similarity)

XYZ size compares sorted axis lengths, so model-space translation and an equivalent model orientation do not matter. A candidate without a recognized solid, surface, body, or mesh receives zero geometry credit even if it copied convincing 2D projections into paper space. Planar REGION no longer counts as true 3D geometry.

The Layout1 foreground is cropped to its occupied bounds and normalized to a 128-by-128 mask before comparison with a four-pixel tolerance. This removes irrelevant page margins and framing while retaining missing, extra, and wrong view content. Presentation scoring requires a populated Layout1 and compares normalized viewport directions after de-duplicating support viewports created by native DRAWINGVIEW objects. A flattened paper-space presentation containing real paper curves is representation-equivalent: when it has fewer than two viewports, camera matching is non-applicable and the trusted Layout1 render remains the view evidence.

Inspection schema v4 isolates genuine 3D bounds. Historical schema-v3 metadata combined them with hidden AutoCAD drawing-view helpers, frequently producing false extents around −10,000. For schema v3, 3D extent similarity is therefore non-applicable instead of a false zero. Evaluator v5/schema v4 enables the physical-size component after its gold cache is rebuilt.

Score JSON records values plus explicit applicability flags, counts, evaluator version, and scorer-config hash. linework_similarity and extent_similarity remain compatibility aliases for registered visual and geometry-extent similarity.

Legacy raster comparison (v1)

The evaluator renders candidate and gold using the same fixed canvas and fit-to-extents rules. Python converts both images to foreground masks, applies a two-pixel tolerance, and computes a symmetric precision/recall F1 score. This path remains only for 2D entries with no sampled vector geometry.

World-space size is scored separately:

width_ratio  = min(candidate_width / gold_width, gold_width / candidate_width)
height_ratio = min(candidate_height / gold_height, gold_height / candidate_height)
extent_similarity = sqrt(width_ratio * height_ratio)

The legacy combined score was:

score = 0.8 * linework_similarity + 0.2 * extent_similarity

Translation was ignored because the evaluator fit model-space extents to the canvas. This metric is retained only for loading and reproducing historical v1 results. It is not used when vector features are present because annotation styles could change plot extents and cause nearly corresponding geometry to miss the two-pixel tolerance.

The scorer emits a diagnostic PNG: matching foreground is black, gold-only foreground is red, and candidate-only foreground is blue. Binary dilation uses an integral image, while vector nearest-neighbor work is spatially indexed and sample-capped. Dense or radically wrong-scale drawings no longer trigger quadratic stalls.

Determinism and version gates

Candidate and gold results must have identical evaluator versions and render-configuration hashes. Gold cache PNG and metadata checksums are validated on every load. The scorer configuration has its own hash, and repeated scoring of identical inputs produces identical result JSON.