BUG: krippendorff_alpha_combined rises with num_scorer_trials even when the scorer's answers do not change #4129

Describe the bug

HarmScorerEvaluator._compute_metrics computes krippendorff_alpha_combined on
np.concatenate((all_human_scores, all_model_scores)), so each scorer trial counts as one more rater.
Every harm entry in the shipped registry has one human rater and 3 trials, so 3 of the 4 rows are the
model agreeing with itself. The value therefore goes up with num_scorer_trials even when
the scorer's answers, and its MAE, stay the same. A scorer that ignores the response gets 0.09 with
1 trial and 0.83 with 10.

This is the number the scorer-metrics doc calls "overall agreement between humans and model", that the
GUI registry shows as "Agreement" for float-scale scorers, and that pretty.py colors green at 0.8 and above.

Steps/Code to Reproduce

from unittest.mock import MagicMock

import numpy as np

from pyrit.score import FloatScaleScorer
from pyrit.score.scorer_evaluation.scorer_evaluator import HarmScorerEvaluator

rng = np.random.default_rng(0)
levels = np.array([0.0, 0.25, 0.5, 0.75, 1.0])
n = 80
human = levels[rng.integers(0, 5, n)][None, :]  # one human rater, like the registry's harm gold sets
scorer_output = levels[rng.integers(0, 5, n)]  # ignores the response; same answer on every trial

evaluator = HarmScorerEvaluator(MagicMock(spec=FloatScaleScorer))
for trials in (1, 3, 5, 10):
    m = evaluator._compute_metrics(
        all_human_scores=human,
        all_model_scores=np.tile(scorer_output, (trials, 1)),
        num_scorer_trials=trials,
    )
    print(f"num_scorer_trials={trials:>2}  MAE={m.mean_absolute_error:.3f}  "
          f"krippendorff_alpha_combined={m.krippendorff_alpha_combined:.3f}")

Expected Results

A human-vs-scorer agreement number that does not depend on how many times the scorer was run
(the MAE is the same on every row).

Actual Results

num_scorer_trials= 1  MAE=0.381  krippendorff_alpha_combined=0.086
num_scorer_trials= 3  MAE=0.381  krippendorff_alpha_combined=0.532
num_scorer_trials= 5  MAE=0.381  krippendorff_alpha_combined=0.686
num_scorer_trials=10  MAE=0.381  krippendorff_alpha_combined=0.828

With the real labels in exploits.csv and a scorer that is one Likert step low on 60% of the
responses (same answer every trial), the value goes from 0.869 (1 trial) to 0.932 (3) and 0.974 (10).

Possible fix

Add a field such as krippendorff_alpha_gold_model: alpha on two rows, the gold label and the
per-response median model score. That is the same pair mean_absolute_error compares, and it does not
depend on the trial count (0.086 on every row above). krippendorff_alpha_combined could stay for the
existing registry entries, with its docstring saying it depends on num_scorer_trials; the GUI
"Agreement" column and pretty.py would switch to the new field, showing None for older entries the
way baseline_mean_absolute_error does.

If you would rather keep the human raters separate, the human rows plus one median-model row also
removes the dependence (it is the same number when there is one rater).

Related: _should_skip_evaluation reuses an entry that has more trials than requested, so a 1-trial
request can get back a 3-trial krippendorff_alpha_combined.

I would like to take this if the direction works for you.

Versions

• OS: macOS 27.2 (arm64)
• Python: 3.11.18
• PyRIT: main @ a97f359 (1.2.0.dev0), editable install
• numpy 2.4.6, scipy 1.17.1