Standardize Task Cost Matrix #188

Activity

1. PGijsbers

   added this to Roadmap (high level)on Apr 26, 2023
2. PGijsbers

   converted this from a draft issue on Sep 2, 2024
3. rohansen856 commented on Nov 18, 2025

   

   rohansen856

   on Nov 18, 2025

   Contributor

   More actions

   Hi @PGijsbers i would like to solve the issue with the following approach:

   Define Cost Matrix Validation Rules

   As i undestand, the Valid Cost Matrix Format is:

   • It Must be a JSON array representing a square matrix
   • All values must be numeric (can be converted to float)
   • Empty array [] is allowed (represents no cost matrix)
   • For Example some valid formats are:
     • [] (no cost matrix)
     • [[0]] (1x1 matrix)
     • [[0, 1], [2, 0]] (2x2 matrix)

   And in case of Invalid Formats:

   • String values: "yes", "adam", "1", "true", "false"
   • Non-JSON format: plain text values
   • Non-square matrices: [[0, 1], [2]]
   • Non-numeric values: [["a", "b"], ["c", "d"]]

   To solve this, we need to implement the following components:

   1. Schema Validation (src/schemas/tasks.py)

   • CostMatrix Pydantic model with strict validation
   • validate_cost_matrix_input() function for database values
   • TaskInput model with input-specific validation

   2. Runtime Validation (src/routers/openml/tasks.py)

   • Added validation at task retrieval time
   • Invalid cost matrices fallback to empty array []
   • Graceful error handling to prevent API breaks

   3. Validation Utilities (src/core/task_validation.py)

   • validate_task_input() - extensible validation framework
   • get_invalid_cost_matrices() - identify problematic entries
   • clean_invalid_cost_matrices() - automated cleanup

   Also, one solution for processing the already present invalid formats:

   A little Database Migration for:

   • Identifying all invalid cost matrix entries
   • Replacing invalid values with empty arrays

   Also implementing something like a --check flag (audit invalid cost matrices) or a --fix flag (automated cleanup of invalid entries) would be good in this case...

   So, In short,

   For Existing Invalid Data:

   1. Run audit using --check or --fix
   2. Apply migration to fix those data points after confirming

   For New Data:

   • Validation runs automatically on task retrieval
   • Invalid values fallback to empty arrays
   • API remains stable during transition

   I would like to work using the above implementation, solving the issue. Thanks!