Evaluate PR #1202 @ b4fba51 #11151
evaluation.yml
on: workflow_dispatch
gate
7s
pr-status
fork-pr-status
Matrix: evaluate / vally-evaluate
report-status
3s
comment-on-pr
20s
deploy-dashboard
Annotations
1 error, 35 warnings, and 43 notices
|
publish-session-data
Process completed with exit code 128.
|
|
evaluate / vally (dotnet-maui--claude-sonnet-5)
vally experiment run reported failures for dotnet-maui (exit 1); adapting completed trial output
|
|
evaluate / vally (dotnet-maui--gpt-5.6-luna)
vally experiment run reported failures for dotnet-maui (exit 1); adapting completed trial output
|
|
evaluate / vally (dotnet-maui--gpt-5.6-luna)
Copilot PAT pool entry 7 has unusable credentials or belongs to a disabled organization; quarantining it and trying another entry
|
|
evaluate / vally (dotnet-test--shard-b--claude-sonnet-5)
vally experiment run reported failures for dotnet-test (exit 1); adapting completed trial output
|
|
evaluate / vally (dotnet-test--shard-b--claude-sonnet-5)
Copilot PAT pool entry 4 has unusable credentials or belongs to a disabled organization; quarantining it and trying another entry
|
|
evaluate / vally (dotnet-test--shard-c--claude-sonnet-5)
vally experiment run reported failures for dotnet-test (exit 1); adapting completed trial output
|
|
evaluate / vally (dotnet-test--shard-c--claude-sonnet-5)
Copilot PAT pool entry 4 has unusable credentials or belongs to a disabled organization; quarantining it and trying another entry
|
|
evaluate / vally (dotnet-test--shard-b--gpt-5.6-luna)
vally experiment run reported failures for dotnet-test (exit 1); adapting completed trial output
|
|
evaluate / vally (dotnet-test--shard-b--gpt-5.6-luna)
Copilot PAT pool entry 4 has unusable credentials or belongs to a disabled organization; quarantining it and trying another entry
|
|
evaluate / vally (dotnet-test--shard-execution--claude-sonnet-5)
vally experiment run reported failures for dotnet-test (exit 1); adapting completed trial output
|
|
evaluate / vally (dotnet-test--shard-default--claude-sonnet-5)
vally experiment run reported failures for dotnet-test (exit 1); adapting completed trial output
|
|
evaluate / vally (dotnet-test--shard-c--gpt-5.6-luna)
vally experiment run reported failures for dotnet-test (exit 1); adapting completed trial output
|
|
evaluate / vally (dotnet-test--shard-generation--claude-sonnet-5)
vally experiment run reported failures for dotnet-test (exit 1); adapting completed trial output
|
|
evaluate / vally (dotnet-test--shard-execution--gpt-5.6-luna)
vally experiment run reported failures for dotnet-test (exit 1); adapting completed trial output
|
|
evaluate / vally (dotnet-test--shard-mstest--claude-sonnet-5)
vally experiment run reported failures for dotnet-test (exit 1); adapting completed trial output
|
|
evaluate / vally (dotnet-test--shard-generation--gpt-5.6-luna)
vally experiment run reported failures for dotnet-test (exit 1); adapting completed trial output
|
|
evaluate / vally (dotnet-test--shard-risk-coverage--claude-sonnet-5)
vally experiment run reported failures for dotnet-test (exit 1); adapting completed trial output
|
|
evaluate / vally (dotnet-test--shard-mstest--gpt-5.6-luna)
vally experiment run reported failures for dotnet-test (exit 1); adapting completed trial output
|
|
evaluate / vally (dotnet-test--shard-mstest--gpt-5.6-luna)
Copilot PAT pool entry 7 has unusable credentials or belongs to a disabled organization; quarantining it and trying another entry
|
|
evaluate / vally (dotnet-test--shard-risk-crap--claude-sonnet-5)
vally experiment run reported failures for dotnet-test (exit 1); adapting completed trial output
|
|
evaluate / vally (dotnet-test--shard-risk-crap--claude-sonnet-5)
Copilot PAT pool entry 7 has unusable credentials or belongs to a disabled organization; quarantining it and trying another entry
|
|
evaluate / vally (dotnet-test--shard-risk-crap--claude-sonnet-5)
Copilot PAT pool entry 4 has unusable credentials or belongs to a disabled organization; quarantining it and trying another entry
|
|
evaluate / vally (dotnet-test--shard-risk-crap--gpt-5.6-luna)
vally experiment run reported failures for dotnet-test (exit 1); adapting completed trial output
|
|
evaluate / vally (dotnet-test--shard-risk-coverage--gpt-5.6-luna)
vally experiment run reported failures for dotnet-test (exit 1); adapting completed trial output
|
|
evaluate / vally (dotnet-test--shard-default--gpt-5.6-luna)
vally compare retry reduced errored trials from 1 to 0 without replacing successful judgments
|
|
evaluate / vally (dotnet-test--shard-default--gpt-5.6-luna)
vally compare returned 1 errored trial(s); retrying once
|
|
evaluate / vally (dotnet-test--shard-default--gpt-5.6-luna)
vally experiment run reported failures for dotnet-test (exit 1); adapting completed trial output
|
|
evaluate / vally (dotnet-test--shard-default--gpt-5.6-luna)
Copilot PAT pool entry 7 has unusable credentials or belongs to a disabled organization; quarantining it and trying another entry
|
|
evaluate / vally (dotnet-test--agent.test-quality-auditor--claude-sonnet-5)
Copilot PAT pool entry 4 has unusable credentials or belongs to a disabled organization; quarantining it and trying another entry
|
|
evaluate / vally (dotnet-test--agent.testability-migration--claude-sonnet-5)
Copilot PAT pool entry 4 has unusable credentials or belongs to a disabled organization; quarantining it and trying another entry
|
|
evaluate / vally (dotnet-test--agent.testability-migration--claude-sonnet-5)
Copilot PAT pool entry 7 has unusable credentials or belongs to a disabled organization; quarantining it and trying another entry
|
|
evaluate / vally (dotnet-test--agent.test-quality-auditor--gpt-5.6-luna)
Copilot PAT pool entry 4 has unusable credentials or belongs to a disabled organization; quarantining it and trying another entry
|
|
evaluate / vally (dotnet-test--agent.code-testing-generator--gpt-5.6-luna)
Copilot PAT pool entry 7 has unusable credentials or belongs to a disabled organization; quarantining it and trying another entry
|
|
Session replay telemetry degraded
The auxiliary session-data publisher failed. Evaluation results remain authoritative, but this PR will not include replay links.
|
|
Session telemetry write preflight failed
SKILLS_DATA_TOKEN could not authenticate for a dry-run write to dotnet/skills-data. Session replay publishing is degraded for this PR; evaluation verdicts remain authoritative.
|
|
gate
"The ubuntu-latest label will migrate to Ubuntu 26 beginning October 19, 2026. For more information, see https://github.com/actions/runner-images/issues/14748"
|
|
discover
"The ubuntu-latest label will migrate to Ubuntu 26 beginning October 19, 2026. For more information, see https://github.com/actions/runner-images/issues/14748"
|
|
evaluate / prepare-validator
"The ubuntu-latest label will migrate to Ubuntu 26 beginning October 19, 2026. For more information, see https://github.com/actions/runner-images/issues/14748"
|
|
evaluate / vally (dotnet-maui--claude-sonnet-5)
"The ubuntu-latest label will migrate to Ubuntu 26 beginning October 19, 2026. For more information, see https://github.com/actions/runner-images/issues/14748"
|
|
evaluate / vally (dotnet-maui--gpt-5.6-luna)
claude-haiku-4.5 does not support reasoning effort; retrying its availability probe without --effort
|
|
evaluate / vally (dotnet-maui--gpt-5.6-luna)
"The ubuntu-latest label will migrate to Ubuntu 26 beginning October 19, 2026. For more information, see https://github.com/actions/runner-images/issues/14748"
|
|
evaluate / vally (dotnet-test--shard-b--claude-sonnet-5)
"The ubuntu-latest label will migrate to Ubuntu 26 beginning October 19, 2026. For more information, see https://github.com/actions/runner-images/issues/14748"
|
|
evaluate / vally (dotnet-test--shard-c--claude-sonnet-5)
"The ubuntu-latest label will migrate to Ubuntu 26 beginning October 19, 2026. For more information, see https://github.com/actions/runner-images/issues/14748"
|
|
evaluate / vally (dotnet-test--shard-b--gpt-5.6-luna)
claude-haiku-4.5 does not support reasoning effort; retrying its availability probe without --effort
|
|
evaluate / vally (dotnet-test--shard-b--gpt-5.6-luna)
"The ubuntu-latest label will migrate to Ubuntu 26 beginning October 19, 2026. For more information, see https://github.com/actions/runner-images/issues/14748"
|
|
evaluate / vally (dotnet-test--shard-execution--claude-sonnet-5)
"The ubuntu-latest label will migrate to Ubuntu 26 beginning October 19, 2026. For more information, see https://github.com/actions/runner-images/issues/14748"
|
|
evaluate / vally (dotnet-test--shard-default--claude-sonnet-5)
"The ubuntu-latest label will migrate to Ubuntu 26 beginning October 19, 2026. For more information, see https://github.com/actions/runner-images/issues/14748"
|
|
evaluate / vally (dotnet-test--shard-c--gpt-5.6-luna)
claude-haiku-4.5 does not support reasoning effort; retrying its availability probe without --effort
|
|
evaluate / vally (dotnet-test--shard-c--gpt-5.6-luna)
"The ubuntu-latest label will migrate to Ubuntu 26 beginning October 19, 2026. For more information, see https://github.com/actions/runner-images/issues/14748"
|
|
evaluate / vally (dotnet-test--shard-generation--claude-sonnet-5)
"The ubuntu-latest label will migrate to Ubuntu 26 beginning October 19, 2026. For more information, see https://github.com/actions/runner-images/issues/14748"
|
|
evaluate / vally (dotnet-test--shard-execution--gpt-5.6-luna)
claude-haiku-4.5 does not support reasoning effort; retrying its availability probe without --effort
|
|
evaluate / vally (dotnet-test--shard-execution--gpt-5.6-luna)
"The ubuntu-latest label will migrate to Ubuntu 26 beginning October 19, 2026. For more information, see https://github.com/actions/runner-images/issues/14748"
|
|
evaluate / vally (dotnet-test--shard-mstest--claude-sonnet-5)
"The ubuntu-latest label will migrate to Ubuntu 26 beginning October 19, 2026. For more information, see https://github.com/actions/runner-images/issues/14748"
|
|
evaluate / vally (dotnet-test--shard-generation--gpt-5.6-luna)
claude-haiku-4.5 does not support reasoning effort; retrying its availability probe without --effort
|
|
evaluate / vally (dotnet-test--shard-generation--gpt-5.6-luna)
"The ubuntu-latest label will migrate to Ubuntu 26 beginning October 19, 2026. For more information, see https://github.com/actions/runner-images/issues/14748"
|
|
evaluate / vally (dotnet-test--shard-risk-coverage--claude-sonnet-5)
"The ubuntu-latest label will migrate to Ubuntu 26 beginning October 19, 2026. For more information, see https://github.com/actions/runner-images/issues/14748"
|
|
evaluate / vally (dotnet-test--shard-mstest--gpt-5.6-luna)
claude-haiku-4.5 does not support reasoning effort; retrying its availability probe without --effort
|
|
evaluate / vally (dotnet-test--shard-mstest--gpt-5.6-luna)
"The ubuntu-latest label will migrate to Ubuntu 26 beginning October 19, 2026. For more information, see https://github.com/actions/runner-images/issues/14748"
|
|
evaluate / vally (dotnet-test--shard-risk-crap--claude-sonnet-5)
"The ubuntu-latest label will migrate to Ubuntu 26 beginning October 19, 2026. For more information, see https://github.com/actions/runner-images/issues/14748"
|
|
evaluate / vally (dotnet-test--shard-risk-crap--gpt-5.6-luna)
claude-haiku-4.5 does not support reasoning effort; retrying its availability probe without --effort
|
|
evaluate / vally (dotnet-test--shard-risk-crap--gpt-5.6-luna)
"The ubuntu-latest label will migrate to Ubuntu 26 beginning October 19, 2026. For more information, see https://github.com/actions/runner-images/issues/14748"
|
|
evaluate / vally (dotnet-test--shard-risk-coverage--gpt-5.6-luna)
claude-haiku-4.5 does not support reasoning effort; retrying its availability probe without --effort
|
|
evaluate / vally (dotnet-test--shard-risk-coverage--gpt-5.6-luna)
"The ubuntu-latest label will migrate to Ubuntu 26 beginning October 19, 2026. For more information, see https://github.com/actions/runner-images/issues/14748"
|
|
evaluate / vally (dotnet-test--shard-default--gpt-5.6-luna)
claude-haiku-4.5 does not support reasoning effort; retrying its availability probe without --effort
|
|
evaluate / vally (dotnet-test--shard-default--gpt-5.6-luna)
"The ubuntu-latest label will migrate to Ubuntu 26 beginning October 19, 2026. For more information, see https://github.com/actions/runner-images/issues/14748"
|
|
evaluate / vally (dotnet-test--agent.code-testing-generator--claude-sonnet-5)
"The ubuntu-latest label will migrate to Ubuntu 26 beginning October 19, 2026. For more information, see https://github.com/actions/runner-images/issues/14748"
|
|
evaluate / vally (dotnet-test--agent.test-quality-auditor--claude-sonnet-5)
"The ubuntu-latest label will migrate to Ubuntu 26 beginning October 19, 2026. For more information, see https://github.com/actions/runner-images/issues/14748"
|
|
evaluate / vally (dotnet-test--agent.testability-migration--claude-sonnet-5)
"The ubuntu-latest label will migrate to Ubuntu 26 beginning October 19, 2026. For more information, see https://github.com/actions/runner-images/issues/14748"
|
|
evaluate / vally (dotnet-test--agent.test-quality-auditor--gpt-5.6-luna)
claude-haiku-4.5 does not support reasoning effort; retrying its availability probe without --effort
|
|
evaluate / vally (dotnet-test--agent.test-quality-auditor--gpt-5.6-luna)
"The ubuntu-latest label will migrate to Ubuntu 26 beginning October 19, 2026. For more information, see https://github.com/actions/runner-images/issues/14748"
|
|
evaluate / vally (dotnet-test--agent.testability-migration--gpt-5.6-luna)
claude-haiku-4.5 does not support reasoning effort; retrying its availability probe without --effort
|
|
evaluate / vally (dotnet-test--agent.testability-migration--gpt-5.6-luna)
"The ubuntu-latest label will migrate to Ubuntu 26 beginning October 19, 2026. For more information, see https://github.com/actions/runner-images/issues/14748"
|
|
evaluate / vally (dotnet-test--agent.code-testing-generator--gpt-5.6-luna)
claude-haiku-4.5 does not support reasoning effort; retrying its availability probe without --effort
|
|
evaluate / vally (dotnet-test--agent.code-testing-generator--gpt-5.6-luna)
"The ubuntu-latest label will migrate to Ubuntu 26 beginning October 19, 2026. For more information, see https://github.com/actions/runner-images/issues/14748"
|
|
report-status
"The ubuntu-latest label will migrate to Ubuntu 26 beginning October 19, 2026. For more information, see https://github.com/actions/runner-images/issues/14748"
|
|
publish-session-data
"The ubuntu-latest label will migrate to Ubuntu 26 beginning October 19, 2026. For more information, see https://github.com/actions/runner-images/issues/14748"
|
|
comment-on-pr
"The ubuntu-latest label will migrate to Ubuntu 26 beginning October 19, 2026. For more information, see https://github.com/actions/runner-images/issues/14748"
|
|
publish-token-data
"The ubuntu-latest label will migrate to Ubuntu 26 beginning October 19, 2026. For more information, see https://github.com/actions/runner-images/issues/14748"
|
Artifacts
Produced during runtime
| Name | Size | Digest | |
|---|---|---|---|
|
vally-results-dotnet-maui--claude-sonnet-5
|
3.05 MB |
sha256:7e1c1f6cc6b991bc77f720673e66542225203d2df118a0e4b03e7defd36d48fc
|
|
|
vally-results-dotnet-maui--gpt-5.6-luna
|
5.86 MB |
sha256:0c8d500f364c12e363a237190f091ccb6757be000ad18457d57d1e5f9ca87f32
|
|
|
vally-results-dotnet-test--agent.code-testing-generator--claude-sonnet-5
|
3.02 MB |
sha256:f8d6f97f30592d66349a7784d4975259ddc2c61263f41e7f18897dadc3bb2966
|
|
|
vally-results-dotnet-test--agent.code-testing-generator--gpt-5.6-luna
|
24.6 MB |
sha256:04ce05ec032379bfb9301428e8b5929052c3589defe98f5666be4773266aa9ef
|
|
|
vally-results-dotnet-test--agent.test-quality-auditor--claude-sonnet-5
|
2 MB |
sha256:03419ee865d9a5253964dca82df668bb4c7931d6457a715591a83b86a6f86a50
|
|
|
vally-results-dotnet-test--agent.test-quality-auditor--gpt-5.6-luna
|
3.8 MB |
sha256:def9dbeb228731eca436be077e1c91e95957ca6d89d687770135c21d0fa457b7
|
|
|
vally-results-dotnet-test--agent.testability-migration--claude-sonnet-5
|
1.68 MB |
sha256:560f77e7f6a6d3e13dc88b7c376d3889c20568a33a94b5de392cb0930106827f
|
|
|
vally-results-dotnet-test--agent.testability-migration--gpt-5.6-luna
|
2.94 MB |
sha256:ae6e3723c52044cb970a58202fe62e78207b014d5bcee29e272f0cef30ae498c
|
|
|
vally-results-dotnet-test--shard-b--claude-sonnet-5
|
5.66 MB |
sha256:6b4afb4430f76421c7a219d58b7c200322d6e4aaa47bae3a768a3469fa4c25f8
|
|
|
vally-results-dotnet-test--shard-b--gpt-5.6-luna
|
12.7 MB |
sha256:e080887a8d71184231f6fd57dfa780eba0c72a306e7fa2396f9615dca195f8b6
|
|
|
vally-results-dotnet-test--shard-c--claude-sonnet-5
|
7.21 MB |
sha256:23736d6b85792e95e3021d406d0e6f400efb6f97b4e915024d99cefaef187897
|
|
|
vally-results-dotnet-test--shard-c--gpt-5.6-luna
|
15 MB |
sha256:3c4bd3f3f0d48639dd5ed3b25d1076ca1d4254e630af01c7f3579c3f51bab9f2
|
|
|
vally-results-dotnet-test--shard-default--claude-sonnet-5
|
10.2 MB |
sha256:56983eb21086b2b9528dc264a0f19d6586c12ee393bb13da6aa0bbccd05e9788
|
|
|
vally-results-dotnet-test--shard-default--gpt-5.6-luna
|
26.3 MB |
sha256:ea35673423e4e56d48add9409a64dbede9aae1b0939680ccd6acee176ee26423
|
|
|
vally-results-dotnet-test--shard-execution--claude-sonnet-5
|
4.25 MB |
sha256:b31d0a722dd875d423b4a761bb976ef3b6dfaccd42200929d9b7b725f0c02570
|
|
|
vally-results-dotnet-test--shard-execution--gpt-5.6-luna
|
8.62 MB |
sha256:59d46e8eea5c17b55b5b0c37b7682fd5be1e65c196592a55015d23ea0efa7846
|
|
|
vally-results-dotnet-test--shard-generation--claude-sonnet-5
|
2.41 MB |
sha256:87e0fa9b21a8586c6225eb4a14ef5583d09054e32a18f7aa810c7c508964240a
|
|
|
vally-results-dotnet-test--shard-generation--gpt-5.6-luna
|
7.92 MB |
sha256:1c5ae3b591ce9cb6fe1d13592d0f26a68ef1321f999591b728ae510490ae6ba5
|
|
|
vally-results-dotnet-test--shard-mstest--claude-sonnet-5
|
1.49 MB |
sha256:684206be580c51073d4ccc5f02d905bb35b6f5005226f36354fb86457c499504
|
|
|
vally-results-dotnet-test--shard-mstest--gpt-5.6-luna
|
2.89 MB |
sha256:32b22566b492c888d02b58094f2de84acc04a00dbec673941ae1bd165274ea88
|
|
|
vally-results-dotnet-test--shard-risk-coverage--claude-sonnet-5
|
2.98 MB |
sha256:8750bd9d6283cd7c68a747fba6a577a048f418682b17413df0d7aee7a03dad81
|
|
|
vally-results-dotnet-test--shard-risk-coverage--gpt-5.6-luna
|
3.88 MB |
sha256:9841a97320a40c4ba2559f7b2cfd17ad31842348b9ce920022163b31bb7f4ab1
|
|
|
vally-results-dotnet-test--shard-risk-crap--claude-sonnet-5
|
1.09 MB |
sha256:ae753b3b1c9a5f261f91a450f9221152f7a1ba72b4df3dc82f03bb6cb4758b61
|
|
|
vally-results-dotnet-test--shard-risk-crap--gpt-5.6-luna
|
1.99 MB |
sha256:b54197e220e04066d6fc6165989ffee455f9ebf4107204d335fe39d82903feb9
|
|