Comparable Evaluation Scorecard

Compare baseline and candidate results case by case, keeping regressions and missing coverage visible.

Version: 1.0.0. License: MIT. Status: tested reference baseline; no comparative ranking is claimed.

Contract

Require unique nonempty case IDs within each run. Compare only shared IDs. Return regressions, improvements, unchanged pass/fail IDs, baseline cases missing in candidate, and new candidate IDs. comparable requires identical nonempty case sets. no_regressions is null when coverage is not comparable.

The exact required input fields and types are in input.schema.json. Extra fields are rejected at declared object boundaries. JSON duplicate keys and non-finite numbers are rejected by the CLI. Input strings are case-sensitive unless stated otherwise. Array order is preserved unless the contract specifies sorting.

Use

Unpack the ZIP, enter its directory, and run:

python3 baseline.py < example.input.json
python3 -m unittest -v

Python 3.10 or newer, standard library only. The reference implementation exposes run(input_dict) and accepts one JSON object on standard input. Success returns {"ok": true, "result": ...} with exit code 0. Invalid input returns {"ok": false, "error": "..."} with exit code 1. example.output.json contains the result without the CLI envelope. Import callers receive exceptions for invalid input.

Example input

{
  "baseline": [
    {
      "id": "a",
      "passed": true
    },
    {
      "id": "b",
      "passed": false
    }
  ],
  "candidate": [
    {
      "id": "a",
      "passed": false
    },
    {
      "id": "b",
      "passed": true
    }
  ]
}

Example result

{
  "comparable": true,
  "no_regressions": false,
  "regressions": [
    "a"
  ],
  "improvements": [
    "b"
  ],
  "unchanged_pass": [],
  "unchanged_fail": [],
  "missing": [],
  "new": []
}

Boundaries

Pass/fail values must come from an external evaluator using the same test definitions and environment. This artifact does not grade answers, choose acceptance thresholds, or treat newly added passing tests as compensation for regressions.

This package performs no network calls, executes no submitted commands, and writes no user files. Host applications own authorization, resource budgets, and persistence. Example values are fixtures, not recommended policies or limits.

Acceptance and provenance

cases.json records named expected results and rejection cases. test_baseline.py runs those cases and checks the command-line contract. manifest.json records file digests. The specification, implementation, and evidence are separate files so a replacement implementation can target the same contract.

Original code and examples generated for this Arcomm collection. No third-party code is bundled. Read LICENSE for reuse terms.

Download

Download the file

SHA-256: 6f67c1a20f3a6da7f17b32d5cf913e27fc0554324167207a23d364cc4936cae1

Size: 6878 bytes.

Revisions

Revision history