helpers package

Submodules

helpers.metrics module

helpers.metrics.compute_classification_metrics(data_test)[source]

compute metrics for just classification

Parameters

data_test (dict) – dict data with fields ‘labels’, ‘preds’

Returns

dict with metrics, ‘classifier_all_acc’: classifier accuracy on all data, also returns AUC for preds_proba

Return type

dict

helpers.metrics.compute_coverage_v_acc_curve(data_test)[source]
Parameters

data_test (dict) – dict data with field {‘defers’: defers_all, ‘labels’: truths_all, ‘hum_preds’: hum_preds_all, ‘preds’: predictions_all, ‘rej_score’: rej_score_all, ‘class_probs’: class_probs_all}

Returns

compute_deferral_metrics(data_test_modified) on different coverage levels, first element of list is compute_deferral_metrics(data_test)

Return type

data (list)

helpers.metrics.compute_deferral_metrics(data_test)[source]

_summary_

Parameters

data_test (dict) – dict data with fields ‘defers’, ‘labels’, ‘hum_preds’, ‘preds’

Returns

dict with metrics, ‘classifier_all_acc’: classifier accuracy on all data

Return type

dict

‘human_all_acc’: human accuracy on all data ‘coverage’: how often classifier predicts

helpers.training module

helpers.training.train_deferral_single_model(Method, dataloader_train, dataloader_test, epochs, lr, verbose=True, test_interval=5, include_scheduler=False)[source]
helpers.training.train_single_model(Method, model, fit, dataloader_train, dataloader_test, epochs, verbose=True, test_interval=5)[source]

Method: the method class model: model in method fit: fit method in Method class

helpers.utils module

class helpers.utils.AverageMeter[source]

Bases: object

Computes and stores the average and current value

reset()[source]
update(val, n=1)[source]

_summary_: Updates the average meter with the new value and the number of samples :param val: value :type val: _type_ :param n: Defaults to 1. :type n: int, optional

class helpers.utils.ExpertDatasetTensor(images, targets, exp_preds)[source]

Bases: Dataset

Generic dataset with expert predictions and labels and images

helpers.utils.accuracy(output, target, topk=(1,))[source]

_summary_

Parameters
  • output (tensor) – output of the model

  • target (_type_) – target

  • topk (tuple, optional) – topk. Defaults to (1,).

Returns

accuracy

Return type

float

Module contents