The runtime · Benchmarks
How we measure a body, not a head.
A single-shot accuracy score measures a jar. Perslis is scored on whether work gets reliably done — and stays done.
Most model benchmarks report first-shot accuracy on a fixed test set. That tells you about a head. It says nothing about whether an operational runtime reliably completes a task, avoids shipping a wrong answer, and does so at a cost you'd pay. Perslis is evaluated on four operational metrics, judged by its own symbolic layer, on held-out tasks. We publish the methodology in full; numbers are populated from reproducible live runs, not asserted.
The metrics
Each metric targets a different way a runtime can fail an operator — not just "was the answer right," but "was the work actually finished, was it wrongly declared done, and what did it cost."
1 · Verified completion rate
The fraction of tasks the runtime completes and independently verifies — tests run, invariants held, evidence sealed. A plausible-but-unverified answer scores as a failure, not a partial. This is the headline number, and it is deliberately harder than first-shot accuracy.
2 · passk — reliability under repetition
The fraction of tasks that pass on every one of k independent attempts. A body has to be dependable, not lucky: a runtime that succeeds one time in three is not operational. passk punishes variance the way production does.
3 · False-cutover rate
For legacy reconstruction and migrations: how often the runtime declares a result "equivalent" or "done" when it is not. Targeted at zero. A wrong "yes" that ships is worse than a slow "no," so this metric is weighted accordingly — the symbolic oracle exists precisely to drive it down.
4 · Cost-to-outcome
Tokens and dollars per verified, deployed result — not per API call. This is the metric an operator actually pays, and it rewards routing cheap models at easy steps and expensive ones only where they earn it.
The harness
- Held-out tasks: operational tasks the runtime hasn't seen — code changes, integrations, and legacy reconstructions with a known-good outcome.
- Symbolic judge: the Captain adjudicates pass/fail against encoded rules and sealed evidence, so scoring isn't a model grading itself.
- Model-swept: the same tasks are run across models to show the runtime's contribution is separable from any one model's IQ.
- Reproducible: every reported number ships with the task set, the seeds, and the receipts to re-run it. A benchmark you can't reproduce is marketing.
Results
The harness is in place; the table below is populated from live runs as they complete. We would rather show a blank cell than a number we can't stand behind.
| Metric | Perslis (best-routed) | Single frontier model, no runtime |
|---|---|---|
| Verified completion | pending live run | pending live run |
| pass5 | pending live run | pending live run |
| False-cutover rate | target: 0 | pending live run |
| Cost-to-outcome | pending live run | pending live run |
Why this framing wins
As model IQ commoditizes, first-shot leaderboards converge — everyone's head is roughly as smart. The gap that remains is operational: who reliably finishes, who avoids the confident mistake, who does it affordably. Those are body metrics, and they're the ones an enterprise buys. Perslis is built to be measured exactly where the durable advantage lives.