镜像站点 · 本页由第三方 GitHub 只读镜像提供,非 GitHub 官方站点,不接受任何登录或凭据输入。前往 github.com
Skip to content

refactor(tests): simplify fixtures and eval tooling - #1292

Merged
mldangelo-oai merged 1 commit into
mainfrom
mdangelo/codex/simplify-test-fixtures
Oct 5, 2026
Merged

mldangelo-oai merged 1 commit into
mainfrom
mdangelo/codex/simplify-test-fixtures

Conversation

@mldangelo-oai

Copy link
Copy Markdown
Collaborator

Summary

SDK tests and SastBench tooling repeat fixture bookkeeping, copier callbacks and private compatibility checks. Reuse existing helpers and remove the unused paths while preserving supported behavior. The complete formatted patch removes 54 maintained lines across 21 files: 45 test lines and 9 tooling lines, with production code unchanged.

Changes

  • Share the stateless completed-scan copier directly and remove constant callback forwarding and an unused reexport.
  • Use existing temporary-directory ownership for policy and workflow fixtures; remove a one-use constructor, duplicated return shape and unused fixture option.
  • Reuse JSONL helpers at equivalent consumers while retaining stricter parsing and independent raw-byte regression checks.
  • Inline case-ID formatting, use native Promptfoo result variables, and remove shard-count validation already enforced by the public runner. Remove only the four tests of that obsolete private precondition.

Testing

  • Fresh plugin and SDK builds passed.
  • SDK, eval and MCP TypeScript checks, generated defaults/schema/model checks, SDK formatting and Prettier on all changed files passed.
  • All 11 deterministic triage scripts passed.
  • Focused SDK tests on Bun 1.4.2: 623 passed, 9 skipped across 14 affected or consumer files.
  • Full SDK tests with seed 12345 and default randomized order: 3,557 passed, 62 skipped per mode, covering all 160 test files. Case identities match the baseline except for the four intentionally removed private-validation cases.
  • The initial concurrent seeded run had one native-config test timeout and one subsequent status-143 error. The six-case file passed in isolation, and one serial retry of the failed shard passed with unchanged source and timeout. The default run passed on its first attempt; the initial stall's cause remains undetermined.
  • Two independent final reviews found no actionable issues. The committed source matches the tested and reviewed files.

Risk and rollout

No public CLI, SDK API, production implementation or dependency changes. Public validation, case-ID formatting, fixture permissions and failure ordering remain intact. This is local macOS qualification; native Windows execution and a live Promptfoo CLI evaluation were not rerun. No rollout or migration is required.

Public disclosure review

  • No customer, partner, prospect, or user identities, data, or identifying details are included.
  • No credentials, personal data, private source, scan findings, or nonpublic links or tickets are included.
  • I reviewed the branch name, title, description, commits, changes, comments, logs, screenshots, attachments, and links for public disclosure.

@mldangelo-oai

Copy link
Copy Markdown
Collaborator Author

@codex review

Please review commit e58c756 for behavior changes in the simplified test fixtures, JSONL helper reuse, private shard validation and native Promptfoo result handling.

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 5, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-05T15:18:18.599489Z e58c756 Draft marked ready
🔒 Security Review ✅ Completed 2026-10-05T15:18:31.260536Z e58c756 Draft marked ready
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@mldangelo-oai
mldangelo-oai marked this pull request as ready for review October 5, 2026 15:15
@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Swish!

Reviewed commit: e58c75669e

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@alandelong-oai alandelong-oai left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed the complete actual PR contribution and changed tests with three independent passes, followed by targeted synthesis. Static review only; product tests were not executed.

No supported introduced correctness or security finding remained in this contribution. Source approval does not establish CI, merge readiness, integration or deployment.

@mldangelo-oai
mldangelo-oai merged commit 3950fe6 into main Oct 5, 2026
61 checks passed
@mldangelo-oai
mldangelo-oai deleted the mdangelo/codex/simplify-test-fixtures branch October 5, 2026 17:37
@github-actions github-actions Bot mentioned this pull request Oct 5, 2026
3 tasks done
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants