Model Test

Model Test

The Model Test panel enables the evaluation of model performance across four key dimensions: Performance, Reliability, Robustness, and Resilience.

Initialize the Panel

To create and initialize the Model Test panel, use:

# Load the Experiment and test a model
from modeva import Experiment
exp = Experiment(name='Demo-SimuCredit')
exp.model_test()

Workflow

Step 1: Select Dataset & Model

1. Select a Dataset: The dataset from the dropdown for processing is automatically selected based on the processed dataset of the experiment (e.g., Demo-SimuCredit_md).
2. Set the Data Selection: Choose a data split (e.g., test).
3. Set Select Model: Pick a registered model from the dropdown (e.g., XGBoost).

Step 2: Performance Evaluation

1. Select Evaluation Metric:
   • Choose a task-specific metric (e.g., MSE for regression, AUC for classification).
2. Residual Analysis:
   • Select a feature for the X-axis to visualize prediction residuals.
3. View Outputs:
   • Summary Table: Displays key accuracy metrics.
   • Residual Plot: Visualizes residuals against the selected feature.

Step 3: Reliability Testing

1. Configure Settings:
   • Expected Coverage: Define confidence interval width (e.g., 0.9 for 90% coverage).
   • Worst Ratio: Set the acceptable error threshold (e.g., 0.1).
2. View Outputs:
   • Calibration Plot: Compares predicted vs. actual confidence intervals.
   • Distribution Shift (PSI): Assesses data stability between training and test sets.

Step 4: Robustness Testing

1. Configure Perturbations:
   • Features: Select features to perturb (e.g., Mortgage).
   • Method: Choose quantile (distribution-based) or normal (Gaussian noise).
   • Noise Level: Define perturbation strength (e.g., 0.1).
2. View Outputs:
   • Robustness Plot: Displays performance degradation under noise.
   • Locate Features: Identifies features with the most significant distribution shift on prediction changes after perturbation.
   • Distribution Shift: Click the bar of interest from the PSI bar plot to view the feature distribution shift between base and worst samples.

Step 5: Resilience Testing

1. Configure Settings:
   • Method: Select worst-sample (identifies hard samples) or outer-sample (boundary samples).
   • Worst Ratio: Define the proportion of worst-case samples (e.g., 0.1).
2. View Outputs:
   • Resilience Plot: Displays performance degradation on challenging samples.
   • PSI Plot: Identify features with the most significant distribution shift on performance.
   • Distribution Shift: Click the bar of interest from the PSI bar plot to view the feature distribution shift between base and worst samples.

Step 6: Saving Results

• Click the |register_icon| button to save test results.

This panel provides actionable insights into model behavior under real-world conditions. For advanced analysis, use the linked distribution visualizations to drill into specific features. For more information, refer to the Diagnostic Suite.