BPt.EvalResultsSubset#

class BPt.EvalResultsSubset(evaluator, subjects, subset_name=None)[source]#

This class represents a subset of EvalResults and is returned as a result of calling EvalResults.subset_by().

This class specifically updates values for a subset of val_subjects, which mean only the following attributes are re-calculated / will be different from the source EvalResults

val_subjects
all_val_subjects
preds
scores
mean_scores
weighted_mean_scores

Attributes

all_train_subjects

This parameter stores the training subjects / index

all_val_subjects

This parameter stores the validation subjects / index

coef_

This attribute represents the mean coef_ as a numpy array across all folds.

cv

If set to store CV is true, a deepcopy of the passed cv splitter will be stored

estimator

This parameter stores the passed saved, unfitted estimator used in this evaluation.

estimators

If the parameter store_estimators is set to True when calling evaluate(), then this parameter will store the fitted estimator in a list.

feat_names

The features names corresponding to any measures of feature importance, stored as a list of lists, where the top level list represents each fold of cross validation.

feature_importances_

This property stores the mean values across fitted estimators assuming each fitted estimator has a non empty feature_importances_ attribute.

fis_

This property stores the mean value across each fold of the CV for either the coef_ or feature_importance_ parameter.

mean_scores

This parameter stores the mean scores as a dictionary of values, where each dictionary is indexed by the name of the scorer, and the dictionary value is the mean score for that scorer.

mean_timing

This property stores information on the fit and scoring times, if requested by the original call to evaluate().

n_folds

A quicker helper property to get the number of CV folds this object was evaluated with.

n_subjects

A quicker helper property to get the sum of the length of train_subjects and val_subjects.

preds

If the parameter store_preds is set to True when calling evaluate(), then this parameter will store the predictions from every evaluate fold.

ps

A saved and pre-processed version of the problem_spec used (with any extra_params applied) when running this instance of Evaluator.

score

This property represents a quick helper for accessing the mean scores of whatever the first scorer is (in the case of multiple scorers).

scores

This property stores the scores for each scorer as a dictionary of lists, where the keys are the names of the scorer and the list represents the score obtained for each fold, where each index corresponds to to a fold of cross validation.

std_scores

This parameter stores the standard deviation scores as a dictionary of values, where each dictionary is indexed by the name of the scorer, and value contains the standard deviation across evaluation folds for that scorer.

timing

This property stores information on the fit and scoring times, if requested by the original call to evaluate().

train_subjects

This parameter stores the training subjects / index

val_subjects

This parameter stores the validation subjects / index

weighted_mean_scores

This property stores the mean scores across evaluation folds (simmilar to mean_scores), but weighted by the number of subjects / datapoints in each fold.

Methods

compare(other[, rope_interval])

This method is designed to perform a statistical comparison between the results from the evaluation stored in this object and another instance of EvalResults.

get_X_transform_df([dataset, fold, ...])

This method is used as a helper for getting the transformed input data for one of the saved models run during evaluate.

get_coefs()

This function returns each coef_ value across fitted estimators.

get_feature_importances()

This function returns each feature_importances_ value across fitted estimators.

get_fis([mean, abs])

This method will return a pandas DataFrame with each row a fold, and each column a feature if the underlying model supported either the coef_ or feature_importance_ parameters.

get_inverse_fis([fis])

Try to inverse transform stored feature importances (either beta weights or automatically calculated feature importances) to their original space.

get_preds_dfs([drop_nan_targets])

This function can be used to return the raw predictions made during evaluation as a list of pandas Dataframes.

permutation_importance([dataset, n_repeats, ...])

This function computes the permutation feature importances from the base scikit-learn function sklearn.inspection.permutation_importance()

run_permutation_test([n_perm, dataset, ...])

Compute signifigance values for the original results according to a permutation test scheme.

subset_by(group[, dataset, decode_values])

Generate instances of EvalResultsSubset based on subsets of subjects based on different unique groups.

to_pickle(loc)

Quick helper to save as pickle.