Welcome to MED3pa documentation!
Overview
Use and Application of the package
The MED3pa
package is specifically designed to address critical challenges in deploying machine learning models, particularly focusing on the robustness and reliability of models under real-world conditions.
It provides comprehensive tools for evaluating model stability and performance in the face of covariate shifts, and problematic data profiles.
Key functionalities
Covariate Shift Detection: Utilizing the detectron subpackage, MED3pa can identify significant shifts in data distributions that might affect the model’s predictions. This feature is crucial for applications such as healthcare, where early detection of shifts can prevent erroneous decisions.
Uncertainty and Confidence Estimation: Through the med3pa subpackage, the package measures the uncertainty and predictive confidence at both individual and group levels. This helps in understanding the reliability of model predictions and in making informed decisions based on model outputs.
Identification of Problematic Profiles: MED3pa analyzes data profiles that consistently lead to poor model performance. This capability allows developers to refine training datasets or retrain models to handle these edge cases effectively.
Software Engineering Principles
The MED3pa
package is constructed with a strong emphasis on software engineering principles, making it a robust and scalable solution for machine learning model evaluation:
Modular Design: The package is structured into four distinct subpackages (detectron, med3pa, models, and datasets), each focusing on different aspects of model training and evaluation. This modularity allows users to easily integrate and utilize specific functionalities tailored to their needs without dealing with unnecessary complexities.
Extensibility: Thanks to its modular architecture, the package can be effortlessly extended to include more functionalities or adapt to different use cases. New models, methods, or data handling procedures can be added with minimal impact on the existing system structure.
Use of Design Patterns: MED3pa employs various design patterns that enhance its maintainability and usability. For example, the use of factory patterns in model creation and strategy patterns in handling different file extensions ensures that the system remains flexible and adaptable to new requirements.
Subpackages
The package is structured into four distinct subpackages : datasets
, models
, detectron
and med3pa
, each focusing on different aspects of the package goals.
This modularity allows users to easily integrate and utilize specific functionalities tailored to their needs without dealing with unnecessary complexities.
Subpackages
- datasets subpackage
- Overview
- loading_context module
- loading_strategies module
- manager module
DatasetsManager
DatasetsManager.combine()
DatasetsManager.get_column_labels()
DatasetsManager.get_dataset_by_type()
DatasetsManager.get_info()
DatasetsManager.reset()
DatasetsManager.reset_datasets()
DatasetsManager.save_dataset_to_csv()
DatasetsManager.set_column_labels()
DatasetsManager.set_from_data()
DatasetsManager.set_from_file()
DatasetsManager.summarize()
- masked module
MaskedDataset
MaskedDataset.clone()
MaskedDataset.combine()
MaskedDataset.get_confidence_scores()
MaskedDataset.get_file_path()
MaskedDataset.get_info()
MaskedDataset.get_observations()
MaskedDataset.get_pseudo_labels()
MaskedDataset.get_pseudo_probabilities()
MaskedDataset.get_sample_counts()
MaskedDataset.get_true_labels()
MaskedDataset.refine()
MaskedDataset.reset_indices()
MaskedDataset.sample_random()
MaskedDataset.sample_uniform()
MaskedDataset.save_to_csv()
MaskedDataset.set_confidence_scores()
MaskedDataset.set_file_path()
MaskedDataset.set_pseudo_labels()
MaskedDataset.set_pseudo_probs_labels()
MaskedDataset.summarize()
MaskedDataset.to_dataframe()
- models subpackage
- Overview
- factories module
- abstract_models module
ClassificationModel
Model
Model.model
Model.model_class
Model.params
Model.data_preparation_strategy
Model.pickled_model
Model.evaluate()
Model.get_data_strategy()
Model.get_info()
Model.get_model()
Model.get_model_type()
Model.get_params()
Model.get_path()
Model.is_pickled()
Model.print_evaluation_results()
Model.save()
Model.set_data_strategy()
Model.set_file_path()
Model.set_model()
Model.set_params()
Model.update_params()
Model.validate_params()
RegressionModel
- concrete_classifiers module
- concrete_regressors module
- abstract_metrics module
- classification_metrics module
ClassificationEvaluationMetrics
ClassificationEvaluationMetrics.accuracy()
ClassificationEvaluationMetrics.average_precision()
ClassificationEvaluationMetrics.balanced_accuracy()
ClassificationEvaluationMetrics.f1_score()
ClassificationEvaluationMetrics.get_metric()
ClassificationEvaluationMetrics.log_loss()
ClassificationEvaluationMetrics.matthews_corrcoef()
ClassificationEvaluationMetrics.npv()
ClassificationEvaluationMetrics.ppv()
ClassificationEvaluationMetrics.precision()
ClassificationEvaluationMetrics.recall()
ClassificationEvaluationMetrics.roc_auc()
ClassificationEvaluationMetrics.sensitivity()
ClassificationEvaluationMetrics.specificity()
ClassificationEvaluationMetrics.supported_metrics()
- regression_metrics module
- data_strategies module
- base module
- detectron subpackage
- Overview
- ensemble module
- record module
DetectronRecord
DetectronRecordsManager
DetectronRecordsManager.count_quantile()
DetectronRecordsManager.counts()
DetectronRecordsManager.freeze()
DetectronRecordsManager.get_evaluation()
DetectronRecordsManager.get_record()
DetectronRecordsManager.load()
DetectronRecordsManager.predicted_probabilities()
DetectronRecordsManager.rejected_count_quantile()
DetectronRecordsManager.rejected_counts()
DetectronRecordsManager.rejection_rates()
DetectronRecordsManager.save()
DetectronRecordsManager.seed()
DetectronRecordsManager.set_evaluation()
DetectronRecordsManager.update()
- stopper module
- strategies module
- experiment module
- comparaison module
- med3pa subpackage
- Overview
- uncertainty module
- models module
APCModel
IPCModel
IPCModel.default_params
IPCModel.evaluate()
IPCModel.get_info()
IPCModel.load_model()
IPCModel.optimize()
IPCModel.predict()
IPCModel.save_model()
IPCModel.supported_ipc_models()
IPCModel.supported_models_params()
IPCModel.supported_regressors_mapping
IPCModel.supported_regressos_params
IPCModel.train()
IPCModel.underlying_models_mapping
MPCModel
- tree module
- Profiles module
- MDR module
- experiment module
- compraison module
Med3paComparison
Med3paComparison.compare_config()
Med3paComparison.compare_experiments()
Med3paComparison.compare_global_metrics()
Med3paComparison.compare_models_evaluation()
Med3paComparison.compare_profiles_detectron_results()
Med3paComparison.compare_profiles_metrics()
Med3paComparison.identify_shared_profiles()
Med3paComparison.is_comparable()
Med3paComparison.save()
Tutorials
Welcome to the tutorials section of the med3pa documentation. Here, we offer comprehensive, step-by-step guides to help you effectively utilize the various subpackages within med3pa
.
Each tutorial is designed to enhance your understanding of the package’s capabilities and to provide practical examples of how to integrate these tools into your data science and machine learning workflows.
Explore each tutorial to learn how to make the most of MED3pa’s robust functionalities for your projects.