[Chen Yufeiyang (feiiiiii5)]

Description

HarmScorerMetrics.krippendorff_alpha_combined concatenates human and model scores, so each scorer trial acts as an extra rater. A scorer that ignores the response rises from 0.086 at 1 trial to 0.828 at 10 trials while MAE stays 0.381.

• Add krippendorff_alpha_gold_model: a two-row alpha between the gold (median human) label and the per-response median model score, so it is invariant to num_scorer_trials.
• Keep krippendorff_alpha_combined unchanged for existing registry data and show the new value in pretty.py.
• The regression reproduces the issue's 1/3/5/10-trial example; it fails on main and passes here. tests/unit/score/test_scorer_evaluator.py 121 passed; related score and output tests 854 passed; Ruff check/format clean.

Closes #4129.

[Copilot]

🟡 Changes recommended

Floating-point noise in median scores can produce a negative ordinal alpha for tolerance-equivalent labels, reporting disagreement even when normalized MAE is zero.

1 open issue

• Floating-point median noise yields negative ordinal alpha · New

What changed in this PR

Adds gold-vs-model agreement to harm scorer evaluation without inflating agreement when identical scorer trials are repeated, addressing #4129 while preserving the existing combined metric.

Changes:

• Computes ordinal alpha between median human labels and median model scores.
• Adds the optional metric to stored results and displays it in pretty output.
• Adds regression coverage for repeated trials and perfect agreement.

File	Description
tests/​unit/​score/​test_scorer_evaluator.py	Tests trial invariance and perfect agreement.
pyrit/​score/​scorer_evaluation/​scorer_metrics.py	Defines and documents the optional metric.
pyrit/​score/​scorer_evaluation/​scorer_evaluator.py	Computes gold-vs-model alpha from two median-score rows.
pyrit/​output/​scorer/​pretty.py	Displays the new metric with quality coloring.

_{Review effort: Balanced}

[Chen Yufeiyang (feiiiiii5)]

Fixed in 4d49108d: the two reliability rows now use gold_scores + diff after the existing diff noise normalization, so a median of 0.2 and 0.4 (0.30000000000000004) matches gold 0.3 exactly instead of reporting a negative ordinal alpha. Added the four-response 0.3 / 0.2+0.4 regression; it fails before the alignment and passes now. tests/unit/score/test_scorer_evaluator.py 122 passed; Ruff check/format clean.

[Chen Yufeiyang (feiiiiii5)]

Thanks for the thorough review and repros Roman Lutz (@romanlutz)! Addressed in 4523853b:

• Global ULP normalization: Ordinal categories are merged across both rows within one ULP from the group representative, resolving cross-item collisions without distorting exact values.
• Cross-item regressions: Added your test cases (cross-item noise yielding 0.8713 and exact model values yielding 0.3832), plus unchained adjacent checks.
• Nits & Pretty output: Bound metrics_by_trials[10] explicitly in the trial invariance test, and added test_pretty.py coverage verifying Krippendorff Alpha (Gold vs Model) renders.

Validation: 157 passed in test_scorer_evaluator.py and test_pretty.py; Ruff clean. (The Windows 3.14 CI failure is the known test_staged_media.py lock flake: PermissionError on temporary path, unrelated to scorer metrics.)

[Qin Dai (Allan-dq)]

I filed #4129, so I checked this against it on 4523853:

• The issue's repro now gives krippendorff_alpha_gold_model = 0.086 at 1, 3, 5 and 10 trials.
• Roman Lutz (@romanlutz)'s two cross-item cases give 0.8713 and 0.3832.
• With two raters, where the gold label can be a midpoint, it matches krippendorff_alpha run on the two
  rows directly (0.9642 on a small example).
• On violence_multi_score.csv (3 raters, 187 responses) with independent per-trial noise, the new alpha
  still goes up with trials (0.951 at 1, 0.986 at 3, 1.000 at 10) and MAE falls with it (0.040, 0.012,
  0.000). I think that is correct: the median of more noisy trials really is closer to gold. One line in
  the docstring could say so, so nobody reads it as the old bug.

Not covered here, and I'm happy to do these as a follow-up once this merges, if you'd like:

• the scorer-metrics doc page (doc/code/scoring/4_scorer_metrics.py) still describes
  krippendorff_alpha_combined as "overall agreement between humans and model" and prints it as the
  scorer's alpha;
• the GUI registry's "Agreement" column and its tooltip (ScorerRegistry.tsx, scorerPresentation.ts)
  still show the combined value;
• the krippendorff_alpha_combined docstring doesn't mention that it depends on num_scorer_trials.

I had offered to take #4129 when I filed it; since this is already well into review, I'm happy for it to
land here.

[Roman Lutz (romanlutz)]

Re-reviewed at 4523853 and against latest main. All review points are addressed, and the one-ULP category merge is a better solution than the decimal rounding I suggested: it fixes both repros (verified: 0.8713 and 0.3832) while preserving genuinely distinct values as close as 1e-10. Trial-invariance holds (0.0857 at 1/3/5/10 trials on the #4129 scenario), and median roundoff never exceeded one ULP across scans of all midpoint pairs on 0.01/0.05/0.1/0.25 grids plus 20k random 4- and 6-rater medians, so the tolerance is sufficient in practice. One known residual: a level with median noise in both directions only partially merges (constructed case: 0.910 vs ideal 1.0) — contrived inputs, small and conservative error, fine to ship.

Qin Dai (@Allan-dq) thanks for the independent verification. I reproduced the direction of your violence_multi_score.csv experiment (MAE 0.076 -> 0.004 while alpha 0.917 -> 0.996 with independent per-trial noise) and agree it's correct behavior: the median of more independent trials genuinely converges to gold, unlike the old artifact where MAE stayed flat. I pushed your suggested docstring note directly (4a90b99) along with a merge of latest main (50e0155); 188 tests pass on the combined state. Your follow-up offer for the docs page, the GUI "Agreement" column/tooltip, and the krippendorff_alpha_combined docstring works for me — please go ahead after this merges.

LGTM, thanks Chen Yufeiyang (@feiiiiii5)!