Repository navigation
Allow for custom evaluation procedures #190
Description
Based on this thread. We should discuss whether we want to allow people to create custom evaluation procedures (e.g., “30 times 3-fold Crossvalidation”) and how we want to support this. Possible concerns include spotty availability (might not support the entire tasktype x procedure matrix) and splintering experimental results. But there are of course also obvious benefits :)
Activity
Jayant-kernel commentedon Mar 15, 2026
@PGijsbers
I’m supporting custom eval procedures, but we should keep comparability.
i thhink
allow custom procedures only as a variant of a known base (e.g., base=10‑fold CV, params: folds/repeats/seed), and whitelist task types that can use it. That keeps leaderboards anchored on the base while still allowing flexibility. Happy to help draft a concrete schema/API if this direction makes sense.