kedro.contrib.io.catalog_with_default package¶
DataCatalogWithDefault
is a DataCatalog
that will fall back to a
default AbstractDataSet
implementation if the requested key in not already
registered in the catalog.
Submodules¶
kedro.contrib.io.catalog_with_default.data_catalog_with_default module¶
A DataCatalog
with a default DataSet
implementation for any data set
which is not registered in the catalog.
-
class
kedro.contrib.io.catalog_with_default.data_catalog_with_default.
DataCatalogWithDefault
(data_sets=None, default=None, remember=False)[source]¶ Bases:
kedro.io.data_catalog.DataCatalog
A
DataCatalog
with a defaultDataSet
implementation for any data set which is not registered in the catalog.-
__init__
(data_sets=None, default=None, remember=False)[source]¶ A
DataCatalog
with a defaultDataSet
implementation for any data set which is not registered in the catalog.Parameters: - data_sets (
Optional
[Dict
[str
,AbstractDataSet
]]) – A dictionary of data set names and data set instances. - default (
Optional
[Callable
[[str
],AbstractDataSet
]]) – A callable which accepts a single argument of type string, the key of the data set, and returns anAbstractDataSet
.load
andsave
calls on data sets which are not registered to the catalog will be delegated to thisAbstractDataSet
. - remember (
bool
) – If True, then store in the catalog anyAbstractDataSet
s provided by thedefault
callable argument. Useful when one want to transition from aDataCatalogWithDefault
to aDataCatalog
: just callDataCatalogWithDefault.to_yaml
, after all required data sets have been saved/loaded, and use the generated YAML file with a newDataCatalog
.
Raises: TypeError
– If default is not a callable.Example:
from kedro.io import CSVLocalDataSet def default_data_set(name): return CSVLocalDataSet(filepath='data/01_raw/' + name) io = DataCatalog(data_sets={}, default=default_data_set) # load the file in data/raw/cars.csv df = io.load("cars.csv")
- data_sets (
-
classmethod
from_config
(catalog, credentials=None, load_versions=None, save_version=None)[source]¶ To create a
DataCatalogWithDefault
from configuration, please use:DataCatalogWithDefault.from_data_catalog( DataCatalog.from_config(catalog, credentials))
Parameters: - catalog (
Optional
[Dict
[str
,Dict
[str
,Any
]]]) – SeeDataCatalog.from_config
- credentials (
Optional
[Dict
[str
,Dict
[str
,Any
]]]) – SeeDataCatalog.from_config
- load_versions (
Optional
[Dict
[str
,str
]]) – SeeDataCatalog.from_config
- save_version (
Optional
[str
]) – SeeDataCatalog.from_config
Raises: ValueError
– If you try to instantiate aDataCatalogWithDefault
- directly with this method.
- catalog (
-
classmethod
from_data_catalog
(data_catalog, default)[source]¶ Convenience factory method to create a
DataCatalogWithDefault
from aDataCatalog
A
DataCatalog
with a defaultDataSet
implementation for any data set which is not registered in the catalog.Parameters: - data_catalog (
DataCatalog
) – TheDataCatalog
to convert to aDataCatalogWithDefault
. - default (
Callable
[[str
],AbstractDataSet
]) – A callable which accepts a single argument of type string, the key of the data set, and returns anAbstractDataSet
.load
andsave
calls on data sets which are not registered to the catalog will be delegated to thisAbstractDataSet
.
Return type: Returns: A new
DataCatalogWithDefault
which contains all theAbstractDataSets
from the provided data-catalog.- data_catalog (
-
load
(name)[source]¶ Loads a registered data set
Parameters: name ( str
) – A data set to be loaded.Return type: Any
Returns: The loaded data as configured. Raises: DataSetNotFoundError
– When a data set with the given name has not yet been registered.
-
save
(name, data)[source]¶ Save data to a registered data set.
Parameters: - name (
str
) – A data set to be saved to. - data (
Any
) – A data object to be saved as configured in the registered data set.
Raises: DataSetNotFoundError
– When a data set with the given name has not yet been registered.- name (
-
shallow_copy
()[source]¶ Returns a shallow copy of the current object. :rtype:
DataCatalogWithDefault
:returns: Copy of the current object.
-