feat!: remove the knowledge track; wp-bench is execution-only - #48
Merged
Conversation
Per maintainer direction: knowledge Q&A tests measure recall, not the ability to produce working WordPress code, and the leaderboard signal should come entirely from code that runs (or fails) in a real WordPress. - Delete the 21 knowledge dataset files (320 tests), knowledge.py, and the knowledge runner loops; load_tests()/filter_tests_by_ids() now work on a flat list of ExecutionTests. - SCORING_VERSION 2.0 -> 3.0: overall is the strict execution pass rate (was 0.3 knowledge + 0.7 execution). Scores are not comparable across this boundary by design. - RESULT_SCHEMA_VERSION 1.0 -> 2.0: the knowledge score key is gone from per-test records; records keep type: "execution" for consumer stability. - run.test_type and run.concurrency are removed from config; both would be silent no-ops now, and extra="forbid" makes old configs fail loudly instead (config means behavior). - Parquet export drops the knowledge rows and the vestigial choices/correct_answer/answer_type columns; the HF loader skips legacy knowledge rows in older exports. - CI dataset job, README, example config, and results notebook updated; usage-capture and record-schema tests ported to execution runs so no execution-side coverage is lost. Verified: 147 tests pass, ruff and mypy clean, and live smoke against the WP 7.0 runtime (dry-run lists 185 execution tests; --check-reference-solution and --check-exploits both work end-to-end). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- Reject zero selected execution tests at the shared selection chokepoint: a missing suite or an execution-less dataset previously let every run mode (including the reference and exploit audits) succeed vacuously, with overall 0.0 beside a null pass rate. - Remove the dead ExecutionTest.test_type discriminator: both loaders hard-coded it and nothing read it, so a caller typo silently became an execution test. - Migrate the results notebook to v3.0 metrics (Runtime replaces Knowledge across the dataframe, charts, radar, and HTML export; the earlier one-line removal left every downstream cell raising KeyError). - Purge the remaining knowledge/test-type references the refactor missed in doc surfaces: the execution-test authoring skill's commands (--test-type no longer exists), datasets/README authoring steps, AGENTS.md layout, and the release checklist row-count step. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- dry-run now routes through the guarded _limit_tests chokepoint, so a missing or execution-less suite errors instead of printing a successful zero count; CLI-level regression test added. - datasets/README drops the knowledge schema section and the dangling knowledge entries in the directory tree. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The guard's ValueError was raised inside _print_dry_run_counts, outside the dry-run try block, so it escaped as a raw traceback. Printing moves inside the handler and the regression test now drives the real CLI via Typer's CliRunner, asserting exit code 1 and the red validation message. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
The following accounts have interacted with this PR and/or linked issues. I will continue to update these lists as activity occurs. You can also manually ask me to refresh this list by adding the If you're merging code through a pull request on GitHub, copy and paste the following into the bottom of the merge commit message. To understand the WordPress project's expectations around crediting contributors, please review the Contributor Attribution page in the Core Handbook. |
Two cleanup agents over the final diff; deletions came back clean, the surviving code had real residuals: - cli.py: collapse the six identical try/except ValueError wrappers into one _run_or_fail helper; the model-override and legacy-single branches merge into a single single-model tail. Drop the two one-line selector wrappers dry-run no longer needs and stop running the stratified selector twice per limited dry-run. - Rename core._limit_tests to select_run_tests: the CLI and tests now consume it across module boundaries, so it is core's public surface. - Fail before booting wp-env: both runners now reject an empty dataset ahead of environment.setup(), so a misconfigured suite errors in milliseconds instead of after a Docker boot. The selection chokepoint guard stays for the general case. - records.py: drop the dead output.answer key (knowledge-only, read by nothing; folded into the already-bumped RESULT_SCHEMA_VERSION 2.0) and the constant type component of the record sort key. - scoring.py: ScoreBreakdown.as_scores_dict() replaces the identical hand-built scores dicts in both runners, keeping the key set next to SCORING_VERSION. - Notebook: null-safe runtime scores (skip_runtime runs). 149 tests, ruff, mypy green; live smoke: dry-run lists 185 tests and a missing suite fails with the clear CLI error before any setup. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This was referenced Aug 5, 2026
Closed
JasonTheAdams
approved these changes
Aug 6, 2026
JasonTheAdams
left a comment
Member
There was a problem hiding this comment.
Thanks for cleaning this up!
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Removes the knowledge Q&A track entirely, per the direction in #wp-bench: the leaderboard signal should come from code that runs (or fails) in a real WordPress, not from recall questions. wp-bench is now execution-only: 185 tests, one track, one score.
knowledge.py, and the knowledge runner loops deleted.load_tests()/filter_tests_by_ids()now work on a flatlist[ExecutionTest].SCORING_VERSION2.0 → 3.0:overallis the strict execution pass rate (was 0.3 knowledge + 0.7 execution). Scores are intentionally not comparable across this boundary.RESULT_SCHEMA_VERSION1.0 → 2.0: theknowledgescore key is gone from per-test records; records keeptype: "execution"for consumer stability.run.test_typeandrun.concurrencyremoved from config. Both would be silent no-ops now (concurrency's only consumer was the knowledge loop; execution parallelism isrun.execution_concurrency). Old configs fail loudly viaextra="forbid"— config means behavior.overall: 0.0beside a null pass rate.ExecutionTest.test_typewas hard-coded by both loaders and read by nothing.choices/correct_answer/answer_typecolumns; the HF loader skips legacy knowledge rows in older exports. CI, README, example config, the authoring skill, datasets/README, AGENTS.md, RELEASE_CHECKLIST, and the results notebook (migrated to Runtime vs Correctness) all updated.Review process
Four adversarial review rounds (independent reviewer) on top of the implementation caught: the vacuous-pass hole, the dead discriminator, a dry-run path that leaked the new error as a raw traceback (now covered by a CliRunner test asserting the red message + exit 1), the notebook's downstream KeyErrors, and stale authoring docs. Final verdict: ship.
Verification
ruffandmypyclean--check-reference-solutionand--check-exploitsboth work end-to-end post-refactorknowledgementions in the tree are intentional (version-bump comments and the legacy-parquet-skip test)🤖 Generated with Claude Code