kedro.contrib.io.catalog_with_default package

DataCatalogWithDefault is a DataCatalog that will fall back to a default AbstractDataSet implementation if the requested key in not already registered in the catalog.

Submodules

kedro.contrib.io.catalog_with_default.data_catalog_with_default module

A DataCatalog with a default DataSet implementation for any data set which is not registered in the catalog.

class kedro.contrib.io.catalog_with_default.data_catalog_with_default.DataCatalogWithDefault(data_sets=None, default=None, remember=False)[source]

Bases: kedro.io.data_catalog.DataCatalog

A DataCatalog with a default DataSet implementation for any data set which is not registered in the catalog.

__init__(data_sets=None, default=None, remember=False)[source]

A DataCatalog with a default DataSet implementation for any data set which is not registered in the catalog.

Parameters:
  • data_sets (Optional[Dict[str, AbstractDataSet]]) – A dictionary of data set names and data set instances.
  • default (Optional[Callable[[str], AbstractDataSet]]) – A callable which accepts a single argument of type string, the key of the data set, and returns an AbstractDataSet. load and save calls on data sets which are not registered to the catalog will be delegated to this AbstractDataSet.
  • remember (bool) – If True, then store in the catalog any AbstractDataSets provided by the default callable argument. Useful when one want to transition from a DataCatalogWithDefault to a DataCatalog: just call DataCatalogWithDefault.to_yaml, after all required data sets have been saved/loaded, and use the generated YAML file with a new DataCatalog.
Raises:

TypeError – If default is not a callable.

Example:

from kedro.io import CSVLocalDataSet

def default_data_set(name):
    return CSVLocalDataSet(filepath='data/01_raw/' + name)

io = DataCatalog(data_sets={},
                 default=default_data_set)

# load the file in data/raw/cars.csv
df = io.load("cars.csv")
classmethod from_config(catalog, credentials=None, load_versions=None, save_version=None)[source]

To create a DataCatalogWithDefault from configuration, please use:

DataCatalogWithDefault.from_data_catalog(
    DataCatalog.from_config(catalog, credentials))
Parameters:
  • catalog (Optional[Dict[str, Dict[str, Any]]]) – See DataCatalog.from_config
  • credentials (Optional[Dict[str, Dict[str, Any]]]) – See DataCatalog.from_config
  • load_versions (Optional[Dict[str, str]]) – See DataCatalog.from_config
  • save_version (Optional[str]) – See DataCatalog.from_config
Raises:
  • ValueError – If you try to instantiate a DataCatalogWithDefault
  • directly with this method.
classmethod from_data_catalog(data_catalog, default)[source]

Convenience factory method to create a DataCatalogWithDefault from a DataCatalog

A DataCatalog with a default DataSet implementation for any data set which is not registered in the catalog.

Parameters:
  • data_catalog (DataCatalog) – The DataCatalog to convert to a DataCatalogWithDefault.
  • default (Callable[[str], AbstractDataSet]) – A callable which accepts a single argument of type string, the key of the data set, and returns an AbstractDataSet. load and save calls on data sets which are not registered to the catalog will be delegated to this AbstractDataSet.
Return type:

DataCatalogWithDefault

Returns:

A new DataCatalogWithDefault which contains all the AbstractDataSets from the provided data-catalog.

load(name)[source]

Loads a registered data set

Parameters:name (str) – A data set to be loaded.
Return type:Any
Returns:The loaded data as configured.
Raises:DataSetNotFoundError – When a data set with the given name has not yet been registered.
save(name, data)[source]

Save data to a registered data set.

Parameters:
  • name (str) – A data set to be saved to.
  • data (Any) – A data object to be saved as configured in the registered data set.
Raises:

DataSetNotFoundError – When a data set with the given name has not yet been registered.

shallow_copy()[source]

Returns a shallow copy of the current object. :rtype: DataCatalogWithDefault :returns: Copy of the current object.