BPt.EvalResultsSubset#

class BPt.EvalResultsSubset(evaluator, subjects, subset_name=None)[source]#

This class represents a subset of EvalResults and is returned as a result of calling EvalResults.subset_by().

This class specifically updates values for a subset of val_subjects, which mean only the following attributes are re-calculated / will be different from the source EvalResults

val_subjects
all_val_subjects
preds
scores
mean_scores
weighted_mean_scores

Attributes

`all_train_subjects`	This parameter stores the training subjects / index
`all_val_subjects`	This parameter stores the validation subjects / index
`coef_`	This attribute represents the mean coef_ as a numpy array across all folds.
`cv`	If set to store CV is true, a deepcopy of the passed cv splitter will be stored
`estimator`	This parameter stores the passed saved, unfitted estimator used in this evaluation.
`estimators`	If the parameter store_estimators is set to True when calling `evaluate()`, then this parameter will store the fitted estimator in a list.
`feat_names`	The features names corresponding to any measures of feature importance, stored as a list of lists, where the top level list represents each fold of cross validation.
`feature_importances_`	This property stores the mean values across fitted estimators assuming each fitted estimator has a non empty feature_importances_ attribute.
`fis_`	This property stores the mean value across each fold of the CV for either the coef_ or feature_importance_ parameter.
`mean_scores`	This parameter stores the mean scores as a dictionary of values, where each dictionary is indexed by the name of the scorer, and the dictionary value is the mean score for that scorer.
`mean_timing`	This property stores information on the fit and scoring times, if requested by the original call to `evaluate()`.
`n_folds`	A quicker helper property to get the number of CV folds this object was evaluated with.
`n_subjects`	A quicker helper property to get the sum of the length of `train_subjects` and `val_subjects`.
`preds`	If the parameter store_preds is set to True when calling `evaluate()`, then this parameter will store the predictions from every evaluate fold.
`ps`	A saved and pre-processed version of the problem_spec used (with any extra_params applied) when running this instance of Evaluator.
`score`	This property represents a quick helper for accessing the mean scores of whatever the first scorer is (in the case of multiple scorers).
`scores`	This property stores the scores for each scorer as a dictionary of lists, where the keys are the names of the scorer and the list represents the score obtained for each fold, where each index corresponds to to a fold of cross validation.
`std_scores`	This parameter stores the standard deviation scores as a dictionary of values, where each dictionary is indexed by the name of the scorer, and value contains the standard deviation across evaluation folds for that scorer.
`timing`	This property stores information on the fit and scoring times, if requested by the original call to `evaluate()`.
`train_subjects`	This parameter stores the training subjects / index
`val_subjects`	This parameter stores the validation subjects / index
`weighted_mean_scores`	This property stores the mean scores across evaluation folds (simmilar to `mean_scores`), but weighted by the number of subjects / datapoints in each fold.

Methods

`compare`(other[, rope_interval])	This method is designed to perform a statistical comparison between the results from the evaluation stored in this object and another instance of `EvalResults`.
`get_X_transform_df`([dataset, fold, ...])	This method is used as a helper for getting the transformed input data for one of the saved models run during evaluate.
`get_coefs`()	This function returns each coef_ value across fitted estimators.
`get_feature_importances`()	This function returns each feature_importances_ value across fitted estimators.
`get_fis`([mean, abs])	This method will return a pandas DataFrame with each row a fold, and each column a feature if the underlying model supported either the coef_ or feature_importance_ parameters.
`get_inverse_fis`([fis])	Try to inverse transform stored feature importances (either beta weights or automatically calculated feature importances) to their original space.
`get_preds_dfs`([drop_nan_targets])	This function can be used to return the raw predictions made during evaluation as a list of pandas Dataframes.
`permutation_importance`([dataset, n_repeats, ...])	This function computes the permutation feature importances from the base scikit-learn function `sklearn.inspection.permutation_importance()`
`run_permutation_test`([n_perm, dataset, ...])	Compute signifigance values for the original results according to a permutation test scheme.
`subset_by`(group[, dataset, decode_values])	Generate instances of `EvalResultsSubset` based on subsets of subjects based on different unique groups.
`to_pickle`(loc)	Quick helper to save as pickle.