command_executed counts commands that were only mentioned, or that failed behind the shell #241

What happened?

command_executed with require_success: true is what the rubric tells authors to use when a command's success is graded (rubric check 1). At a253f34, a successful tool call can also satisfy it when the command was only mentioned, or when the shell hid the command's failure:

1. Another tool only names the command. With the default tool_name: null, the pattern is also searched in json.dumps(parameters) of every non-shell tool (command_executed.py L189, L204, L152). A TodoWrite item "Run uip solution deploy", a Grep for that string, or a Write of a notes file counts as running it. The tool call succeeded, so require_success counts it too. The same text also trips a max_count: 0 guard: a note that says "do not run X" fails the guard.
2. A Bash call only contains the text. echo 'python app.py', # python app.py and true || python app.py match python app\.py, and the call succeeds.
3. The shell hides the failure. require_success reads the reported success of the whole tool call (L193). python app.py 2>&1 | tail -n 5 and python app.py; true succeed when app.py crashes. An agent that pipes output through tail hits this without any intent to cheat.

Impact: the criterion reports that the agent ran the command when it did not, or when the command failed. How much this moves the task score depends on the other criteria. With stop_early: {on_pass: stop}, the false live pass also ends the run, and a run the watcher cut is graded on its armed criteria only. CE034 requires require_success on such criteria to stop a crashed command from passing, but cases 2 and 3 pass with it set.

Steps to reproduce

Criterion level (no model, no network). From a checkout at a253f34 after uv sync --extra dev, run this with .venv/bin/python:

from datetime import datetime
from coder_eval.criteria.command_executed import CommandExecutedChecker
from coder_eval.evaluation.checker import SuccessChecker
from coder_eval.models import CommandExecutedCriterion
from coder_eval.models.results import TurnRecord
from coder_eval.models.telemetry import CommandTelemetry

def score(crit, tool, params):
    call = CommandTelemetry(tool_name=tool, tool_id="t", timestamp=datetime.now(), parameters=params, result_status="success")
    turn = TurnRecord(iteration=1, user_input="p", agent_output="done", commands=[call])
    return SuccessChecker(sandbox=None).check(crit, turn_records=[turn]).score, CommandExecutedChecker().live_verdict(crit, [turn])

deploy = CommandExecutedCriterion(description="deployed", command_pattern=r"uip\s+solution\s+deploy", require_success=True)
for tool, params in [("TodoWrite", {"todos": [{"content": "Run uip solution deploy", "status": "pending"}]}),
                     ("Grep", {"pattern": "uip solution deploy", "path": "."}),
                     ("Write", {"file_path": "NOTES.md", "content": "Next: uip solution deploy"})]:
    print(score(deploy, tool, params), tool)

ran = CommandExecutedCriterion(description="ran app", tool_name="Bash", command_pattern=r"python app\.py", require_success=True)
for cmd in ["echo 'python app.py'", "# python app.py", "true || python app.py",
            "python app.py 2>&1 | tail -n 5", "python app.py; true"]:
    print(score(ran, "Bash", {"command": cmd}), cmd)

Each record is a successful tool call, and every line prints (1.0, 'pass'): score 1.0, and a live pass that can end an early-stop run. The first six calls never run the command. The last two do, but the shell reports them as successful also when app.py crashes; the pipeline run below shows this.

Full pipeline. coder-eval run on the three in-repo tasks/early_stop_*.yaml, with the task's agent replaced by a scripted agent. It executes Bash and Write in the sandbox and records other tool calls as successful no-ops. Task loading, sandbox, early-stop watcher, checker and gating are unchanged. Each task asks for an app.py that prints a greeting, and a run of python app.py.

These tasks are framework examples; the concern is suites built on this criterion. A copy of the first task with the default tool_name gives SUCCESS 1.000 for "write app.py, then a TodoWrite item Run python app.py", with no run at all. I can share the scripted agent (about 170 lines) if it helps.

Environment

coder_eval main at a253f34 (0.12.16 plus one commit), Python 3.13.17, Ubuntu 24.04, default tempdir sandbox.

Suggested fix

• Case 1: with tool_name: null, search a command_pattern only in the command param of a call (the Bash command, or a shell tool under another name). A task that wants another tool's params names that tool. A PR for this follows.
• Cases 2 and 3: the telemetry has one success flag per tool call, so it cannot show whether the matched program itself ran or succeeded. Parsing the shell text could catch simple forms such as echo cmd, but not reliably, and the matcher keeps the raw text on purpose (fix(command-executed): match patterns against shell-normalized commands #77). The same PR documents this next to require_success, in rubric check 1 and in the task skill, and tells authors to also check the effect of the command. A full fix needs per-invocation evidence, for example a wrapper around the real binary that records its exit code.