# Detect Desk > Read benchmark and evaluation results and see what the measurement could actually > detect. A difference you can see is not a difference you can prove: the same 3% > headline is decisive at one spread and meaningless at another, the smallest effect a > suite can resolve is fixed before it runs, k comparisons at alpha alarm on noise with > probability 1-(1-alpha)^k, a p99 from 50 runs IS the maximum, and 80% against 70% is > not separable at 100 tasks each. Free browser-side statistics, five paid lanes. Live at https://detect-desk.skillsafe.ai/ ยท API at https://detect-desk.skillsafe.ai/api.html Derived from the `benchmark` and `agent-eval` skills in `affaan-m/everything-claude-code` (https://github.com/affaan-m/everything-claude-code). Not affiliated with or endorsed by affaan-m. ## The one thing to know **The same headline gives opposite verdicts.** ```text 100ms -> 103ms, 5 runs a side. Reported as "3% slower" either way. sd 0.2 ms t = 23.72 p = <0.0001 interval +2.708 to +3.292 sd 4 ms t = 1.19 p = 0.2697 interval -2.834 to +8.834 ``` The percentage is identical and the spread that decides between them is the one number the report does not carry. **The smallest effect a suite can detect is fixed before it runs.** ```text MDE = (z_alpha/2 + z_power) x cv x sqrt(2/n) at cv 4%, alpha 0.05, power 80% n = 5 detects 7.09% power against 3%: 21.9% n = 30 detects 2.89% power against 3%: 82.8% to detect 3%: 28 runs a side to detect 1%: 252 runs ``` A five-run job has 21.9% power against a 3% regression - four times in five it reports nothing and is believed. `n` enters as a square root, so halving the effect you want to see costs four times the runs. **A suite of k benchmarks alarms on noise, by construction.** ```text P(at least one fires with nothing wrong) = 1 - (1 - alpha)^k k = 1 5% hold the family at 0.05 with alpha = 0.05 k = 5 22.6% hold the family at 0.05 with alpha = 0.0102 k = 10 40.1% hold the family at 0.05 with alpha = 0.0051 k = 20 64.2% hold the family at 0.05 with alpha = 0.0026 k = 40 87.1% hold the family at 0.05 with alpha = 0.0013 k = 100 99.4% hold the family at 0.05 with alpha = 0.00051 ``` At 20 benchmarks a clean tree goes red 64.2% of the time. Holm's step-down thresholds hold the family at alpha and are uniformly more powerful than Bonferroni. **A tail percentile from n runs is an order statistic.** ```text the count below the true quantile is Binomial(n, p), so the rank interval is exact n = 20 p99 could be rank 19-20 <- the maximum n = 50 p99 could be rank 48-50 <- the maximum n = 100 p99 could be rank 97-100 <- the maximum n = 300 p99 could be rank 293-300 <- the maximum ``` From 50 runs the p99 IS the largest run, and the largest of n draws grows with n - so it is not comparable against a week that ran a different number of times. 373 runs is the smallest count at which that stops being true. **And a pass rate is worse.** ```text n = 20 16/20 = 80% interval [58.4%, 91.9%] n = 100 80/100 = 80% interval [71.1%, 86.7%] 80% against 70% at n = 100: z = 1.63, p = 0.1025 NOT separable to separate them at 80% power: 294 tasks per arm ``` An agent leaderboard published on twenty tasks is ordering noise. ## The same headline, six spreads | Run-to-run sd | Which is a cv of | The headline | Welch t | p | The interval on the difference | Verdict | | --- | --- | --- | --- | --- | --- | --- | | 0.2 ms | 0.2% | +3% | 23.72 | <0.0001 | +2.708 ms to +3.292 ms | **separable** | | 0.5 ms | 0.5% | +3% | 9.49 | <0.0001 | +2.271 ms to +3.729 ms | **separable** | | 1 ms | 1% | +3% | 4.74 | 0.0015 | +1.542 ms to +4.458 ms | **separable** | | 2 ms | 2% | +3% | 2.37 | 0.0451 | +0.0831 ms to +5.917 ms | **separable** | | 4 ms | 4% | +3% | 1.19 | 0.2697 | -2.834 ms to +8.834 ms | noise | | 8 ms | 8% | +3% | 0.59 | 0.5696 | -8.668 ms to +14.67 ms | noise | The same measurement every time: 100 ms against 103 ms over 5 runs a side. **Every row reports "3% slower"**, and the verdict flips from noise to certain purely on the spread - which is the one number the headline does not carry. Read the last two columns together: at 0.2 ms the effect is known to within 0.5834 ms, and at 8 ms the interval is 23.34 ms wide and contains a speed-up. ## The detection floor | Runs a side | The smallest effect it can detect | Its power against a 3% effect | Runs to detect 3% | Runs to detect 1% | | --- | --- | --- | --- | --- | | 3 | **9.15%** | 14.9% | 28 | 252 | | 5 | **7.09%** | 21.9% | 28 | 252 | | 10 | **5.01%** | 38.9% | 28 | 252 | | 20 | **3.54%** | 66% | 28 | 252 | | 30 | **2.89%** | 82.8% | 28 | 252 | | 50 | **2.24%** | 96.3% | 28 | 252 | | 100 | **1.58%** | 100% | 28 | 252 | At a 4% coefficient of variation, alpha 0.05 and 80% power. **The first column is chosen and the second follows** - the detection floor is fixed before any measurement is taken, and no amount of looking at the result changes it. The third column is the same fact stated as a probability: five runs a side have 21.9% power against a 3% regression, so four times in five it will report nothing and be believed. And because `n` enters as a square root, **halving the detectable effect costs four times the runs**: 28 to see 3%, 252 to see 1%. ## What a suite of comparisons costs | Comparisons | Chance at least one fires with nothing wrong | False alarms expected | Per-comparison alpha to hold the family at 0.05 | Bonferroni | | --- | --- | --- | --- | --- | | 1 | **5%** | 0.05 | 0.05 | 0.05 | | 5 | **22.6%** | 0.25 | 0.0102 | 0.01 | | 10 | **40.1%** | 0.5 | 0.0051 | 0.005 | | 20 | **64.2%** | 1 | 0.0026 | 0.0025 | | 40 | **87.1%** | 2 | 0.0013 | 0.0013 | | 100 | **99.4%** | 5 | 0.00051 | 0.0005 | `1 - (1 - alpha)^k` at alpha 0.05. **This is a property of the suite's size, not of the code being tested.** A twenty-benchmark job that flags anything at 0.05 produces a red result on a clean tree 64.2% of the time - which is why a suite that alarms most weeks is not evidence of a problem with the code. Use Holm's step-down thresholds: they hold the family at alpha and are uniformly more powerful than the Bonferroni column beside them. ## The tail, as an order statistic | Runs | The p99 could be any of these ranks | Which is | The rank interval spans | Is it the maximum | | --- | --- | --- | --- | --- | | 20 | 19 to 20 of 20 | 95% to 100% | 2 ranks | **yes** | | 50 | 48 to 50 of 50 | 96% to 100% | 3 ranks | **yes** | | 100 | 97 to 100 of 100 | 97% to 100% | 4 ranks | **yes** | | 300 | 293 to 300 of 300 | 97.7% to 100% | 8 ranks | **yes** | | 1000 | 983 to 996 of 1000 | 98.3% to 99.6% | 14 ranks | no | | 3000 | 2959 to 2980 of 3000 | 98.6% to 99.3% | 22 ranks | no | The count of samples below the true quantile is `Binomial(n, p)`, so these intervals are exact and assume nothing about the distribution's shape. **From 50 runs the p99 is the largest run**, and the largest of `n` draws grows with `n` - so that figure is not comparable against a week when the suite ran a different number of times. 373 runs is the smallest count at which the interval stops touching the maximum at all. ## The pass rate, as a proportion | Tasks per arm | 80% reads as | Its interval | Against 70%, z | p | Separable | | --- | --- | --- | --- | --- | --- | | 10 | 8/10 | [49%, 94.3%] | 0.52 | 0.6056 | no | | 20 | 16/20 | [58.4%, 91.9%] | 0.73 | 0.4652 | no | | 50 | 40/50 | [67%, 88.8%] | 1.15 | 0.2482 | no | | 100 | 80/100 | [71.1%, 86.7%] | 1.63 | 0.1025 | no | | 300 | 240/300 | [75.1%, 84.1%] | 2.83 | 0.0047 | **yes** | | 1000 | 800/1000 | [77.4%, 82.4%] | 5.16 | <0.0001 | **yes** | A pass rate is a proportion, and a proportion from `n` trials is far less certain than it looks: 8/10 has an interval of [49%, 94.3%]. **80% against 70% is not separable even at 100 tasks each**, and separating them at 80% power takes 294 tasks per arm. The Wilson interval is used rather than the normal one because the normal interval reports [100%, 100%] for a clean sweep, which is not a claim twenty tasks can support. ## Constants and thresholds | Constant | Value | What it decides | | --- | --- | --- | | `DEFAULT_ALPHA` | 0.05 | the assumed false-alarm rate per comparison, and every verdict with it | | `DEFAULT_POWER` | 80% | the assumed chance of noticing a real effect of the size asked about | | `NOISY_CV` | 5% | a variant this variable between identical runs is worth naming | | `TINY_N` | 5 | fewer runs than this is worth naming on its own | | `BIG_OUTLIER` | 3 sd | how far out a single run has to be before its effect on the mean is worth naming | | `FWER_ALARM` | 25% | a family-wise false-alarm rate past this is worth naming | Only the first two are conventions with a literature; the rest are this page's thresholds for when a difference is worth saying. Every t, p, interval and run count is exact arithmetic and moves with none of them - the thresholds decide only what gets called a warning. ## Sheet grammar A sheet is a header of `KEY: value` lines and one block: `RUNS:` for measurements, `SUMMARY:` for a table that already has n, mean and sd, or `TASKS:` for pass counts. ```text JOB: what this measured (optional, echoed back) UNIT: ms (echoed on every figure; not converted) BETTER: lower | higher (which direction is an improvement; assumed lower) ALPHA: 0.05 (the false-alarm rate per comparison; assumed 0.05) POWER: 0.8 (the chance of noticing a real effect; assumed 0.8) TARGET: 3% (the regression size you need to catch) PERCENTILE: 0.95 (which tail to report; assumed 0.95) RUNS: 102, 98, 105, 99, 101 (comma or space separated; wraps onto indented continuation lines) SUMMARY: n=30 mean=102.4 sd=4.1 (or cv= instead of sd=) TASKS: 17/20 (or passed=17 total=20) ``` **The first variant is the BASELINE and every other is compared against it.** That is why the order is part of the grammar: which variant is the reference decides every delta, every interval and which way round the percentages read. **A `SUMMARY:` block is a first-class input, not a degraded one.** Welch's t needs only n, mean and sd from each side, which is exactly what a benchmark harness's own summary table prints - so pasting it gives the same test, the same interval and the same detection floor. What it cannot give is the tail, the median, or what a single extreme run is doing, all of which need the runs themselves. `TARGET:` is the one line that turns this page from a description into a check: with it, the page can say whether the gate you set is inside what the suite can resolve. Anything the reader cannot place is listed as a problem rather than skipped. A variant that quietly vanished would change the comparison count, the family-wise rate and which variant is the baseline - so a swallowed line would move every figure on the page at once. ## Lanes | Lane | What it produces | | --- | --- | | `plan` | Design the measurement before you run it | | `read` (primary) | Is this difference real, and what could this run have seen | | `power` | The detection floor: what this suite can and cannot see | | `tail` | The shape: percentiles, outliers, and which statistic to quote | | `decide` | Decide what changes: more runs, a quieter machine, or the question | ## A worked sheet ```text JOB: the p95 latency regression CI flagged on PR 4412 UNIT: ms ALPHA: 0.05 POWER: 0.8 TARGET: 3% RUNS: main 96, 104, 99, 105, 96 pr4412 99, 107, 102, 108, 99 ``` | Variant | Runs | Mean | sd | cv | Against the baseline | p | Could it have seen it | | --- | --- | --- | --- | --- | --- | --- | --- | | `main` | 5 | 100 ms | 4.301 ms | 4.3% | _baseline_ | โ€” | โ€” | | `pr4412` | 5 | 103 ms | 4.301 ms | 4.18% | +3% | 0.3022 | **no - floor is 7.62%** | The sheet in the worked example. 2 variants over 10 runs, +3% on `pr4412`, p 0.3022, floor 7.62%. **Read the last two columns together**: the p-value says the run could not separate them, and the column beside it says the run was never able to. ## Findings | Finding | Severity | Scope | What it says | | --- | --- | --- | --- | | `NO-RESULTS-TO-READ` | error | sheet | The sheet declares no runs and no tasks | | `A-SINGLE-RUN-IS-NOT-A-MEASUREMENT` | error | sample | A variant has one run, so it has no spread at all | | `TOO-FEW-RUNS-TO-SAY-ANYTHING` | warning | sample | A variant has too few runs to estimate its own noise | | `A-VARIANT-IS-NOISY` | warning | sample | A variant's run-to-run spread is large relative to its own mean | | `THE-DIFFERENCE-IS-INDISTINGUISHABLE-FROM-NOISE` | warning | effect | The difference is not separable from run-to-run variation | | `THE-INTERVAL-CONTAINS-BOTH-DIRECTIONS` | warning | effect | The interval on the difference spans zero, so even the sign is unknown | | `THE-SUITE-CANNOT-SEE-THE-EFFECT-IT-MEASURED` | warning | power | The effect is smaller than the smallest this suite could detect | | `THE-TARGET-IS-BELOW-THE-DETECTION-FLOOR` | warning | power | The regression threshold you set is smaller than the suite can resolve | | `THE-SUITE-ALARMS-ON-NOISE` | warning | family | With this many comparisons, a false alarm is more likely than not | | `HOLM-CHANGES-THE-VERDICT` | warning | family | A comparison that passes on its own does not survive the family correction | | `THE-TAIL-PERCENTILE-IS-THE-MAXIMUM` | warning | tail | The tail percentile you asked for is, at this sample size, the largest run | | `ONE-POINT-IS-CARRYING-THE-MEAN` | warning | tail | A single extreme run is moving the mean by more than the effect being measured | | `THE-PASS-RATES-ARE-NOT-DISTINGUISHABLE` | warning | rate | Two pass rates that look different are not separable at this task count | | `HOW-MANY-RUNS-EACH` | note | sample | How many runs each variant got, and what that alone decides | | `THE-SPREAD-OF-EACH-VARIANT` | note | sample | Each variant's spread, in its own units and as a share of its mean | | `THE-HEADLINE-DIFFERENCE` | note | effect | The difference, as it would be reported, with what that number leaves out | | `THE-DIFFERENCE-IS-SEPARABLE` | note | effect | The difference is larger than run-to-run variation would produce | | `THE-SMALLEST-EFFECT-THIS-SUITE-CAN-SEE` | note | power | The minimum detectable effect, which is fixed before any measurement | | `THE-RUNS-NEEDED-TO-SETTLE-IT` | note | power | How many runs a side would settle the difference you actually saw | | `THE-FAMILY-FALSE-ALARM-RATE` | note | family | What this many comparisons costs in false alarms, before any result | | `THE-TAIL-IS-AN-ORDER-STATISTIC` | note | tail | Which ranks the reported percentile could actually be | | `THE-MEAN-AND-THE-MEDIAN-DISAGREE` | note | tail | The mean and the median of the same runs, side by side | | `THE-PASS-RATE-INTERVAL` | note | rate | Each pass rate with the interval its task count actually supports | | `THE-TASKS-NEEDED-TO-SEPARATE-THEM` | note | rate | How many tasks per arm would separate the pass rates you saw | | `THE-SHEET-HAS-LINES-THIS-PAGE-COULD-NOT-READ` | warning | sheet | Some lines were not readable | | `THE-ALPHA-WAS-NOT-DECLARED` | note | sheet | The false-alarm rate was assumed rather than stated | | `THE-POWER-WAS-NOT-DECLARED` | note | sheet | The power target was assumed rather than stated | 27 findings: 2 errors, 12 warnings and 13 notes. **Nothing that fires on every usable sheet is a warning.** Every sheet has a run count, every sheet has a spread, every comparison has a detection floor and every suite has a family-wise rate - so those are notes. What gets a warning is a difference that cannot be separated from noise, a suite that could never have seen the effect it reported, a percentile that is really the maximum, or a single run carrying the mean. ## What this page cannot do This page reads a table of results, not your benchmark harness. It never runs anything, never times anything and never sees the machine, so: - **It is the ARITHMETIC on the numbers you paste.** Whether those numbers came from a fair comparison is the harder half and this page cannot see it: a warm cache, a different commit, a noisy neighbour on a shared runner and an interleaved versus batched run order all change what the samples mean, and none of them changes what the arithmetic returns. - **The runs are assumed INDEPENDENT and comparable.** Consecutive runs on one machine usually are not: they drift with temperature, cache state and background load, and the drift makes the spread look smaller within a batch and larger between them. Interleaving the variants is what buys the independence this page assumes. - **`t` assumes the MEAN is roughly normal, which is about `n`, not about the runs.** The central limit theorem does the work here, so the test is robust for a dozen runs of a mildly skewed measurement and not robust for three runs of a heavily skewed one. Latency is heavily skewed; that is what the tail section is for. - **A p-value is not the chance the change did nothing.** It is the chance of seeing a gap at least this large IF nothing had changed. The quantity you usually want is the interval on the difference, which is why it is printed beside every verdict. - **Not separable is not the same as the same.** A run that cannot tell two variants apart has established that, and nothing else - which is why the interval and the detection floor are reported next to it rather than instead of it. - **The family-wise arithmetic assumes the comparisons are independent.** Benchmarks on one machine share their noise, so the true rate is usually a little lower than `1-(1-alpha)^k` - and Holm's thresholds are valid regardless of the dependence, which is another reason to prefer them. - **Nothing here says whether the change is worth making.** It says whether this measurement could have seen it, whether it did, and what it would take to settle. ## API `POST https://api.skillsafe.ai/v1/app-api/run` with an app session token. The body IS the input object โ€” never wrapped in an `input` key. Fields: `task` (one of `plan`, `read`, `power`, `tail`, `decide`), `rules`, `prescan`, plus the lane's own fields. `POST .../estimate` with the same body returns `hold_credits` and costs nothing. ## Provenance Every table and every figure above is generated from `detect.js` by `build-skill.js`. Nothing is typed twice, so nothing can drift.