kedro.contrib.config.templated_config.TemplatedConfigLoader¶
-
class
kedro.contrib.config.templated_config.
TemplatedConfigLoader
(conf_paths, *, globals_pattern=None, globals_dict=None)[source]¶ Bases:
kedro.config.config.ConfigLoader
Extension of the
ConfigLoader
class that allows for template values, wrapped in brackets like: ${…}, to be automatically formatted based on the configs.The easiest way to use this class is by incorporating it into the
KedroContext
. This can be done by extending theKedroContext
and overwriting the config_loader method, making it return aTemplatedConfigLoader
object instead of aConfigLoader
object.For this method to work, the context_path variable in .kedro.yml needs to be pointing at this newly created class. The run.py script has an extension of the
KedroContext
by default, called theProjectContext
.Example:
from kedro.context import KedroContext, load_context from kedro.contrib.config import TemplatedConfigLoader class MyNewContext(KedroContext): def _create_config_loader(self, conf_paths: Iterable[str]) -> TemplatedConfigLoader: return TemplatedConfigLoader( conf_paths, globals_pattern="*globals.yml", globals_dict={"param1": "CSVLocalDataSet"} ) my_context = load_context(Path.cwd(), env=env) my_context.run(tags, runner, node_names, from_nodes, to_nodes)
The contents of the dictionary resulting from the globals_pattern get merged with the
globals_dict
. In case of conflicts, the keys inglobals_dict
take precedence. If the formatting key is missing from the dictionary, the default template value is used (the format is “${key|default value}”). If no default is set, aValueError
will be raised.Global parameters can be namespaced as well. An example could work as follows:
globals.yml
bucket: "my_s3_bucket" environment: "dev" datasets: csv: "CSVS3DataSet" spark: "SparkDataSet" folders: raw: "01_raw" int: "02_intermediate" pri: "03_primary" fea: "04_features"
catalog.yml
raw_boat_data: type: "${datasets.spark}" filepath: "s3a://${bucket}/${environment}/${folders.raw}/boats.csv" file_format: parquet raw_car_data: type: "${datasets.csv}" filepath: "data/${environment}/${folders.raw}/cars.csv" bucket_name: "${bucket}" file_format: "${car_file_format|parquet}"
This uses
jmespath
in the background. For more information see: https://github.com/jmespath/jmespath.py and http://jmespath.org/.Methods
TemplatedConfigLoader.__init__
(conf_paths, *)Instantiate a TemplatedConfigLoader
.TemplatedConfigLoader.get
(*patterns)Tries to resolve the template variables in the config dictionary provided by the ConfigLoader
(super class)get
method using the dictionary of replacement values obtained in the__init__
method.-
__init__
(conf_paths, *, globals_pattern=None, globals_dict=None)[source]¶ Instantiate a
TemplatedConfigLoader
.Parameters: - conf_paths (
Union
[str
,Iterable
[str
]]) – Non-empty path or list of paths to configuration directories. - globals_pattern (
Optional
[str
]) – Optional keyword-only argument specifying a glob pattern. Files that match the pattern will be loaded as a formatting dictionary. - globals_dict (
Optional
[Dict
[str
,Any
]]) – Optional keyword-only argument specifying a formatting dictionary. This dictionary will get merged with the globals dictionary obtained from the globals_pattern. In case of duplicate keys, theglobals_dict
keys take precedence.
- conf_paths (
-
get
(*patterns)[source]¶ Tries to resolve the template variables in the config dictionary provided by the
ConfigLoader
(super class)get
method using the dictionary of replacement values obtained in the__init__
method.Parameters: patterns ( str
) – Glob patterns to match. Files, which names match any of the specified patterns, will be processed.Return type: Dict
[str
,Any
]Returns: A Python dictionary with the combined configuration from all configuration files. Note: any keys that start with _ will be ignored. String values wrapped in ${…} will be replaced with the result of the corresponding JMESpath expression evaluated against globals (see __init for more configuration files. Note: any keys that start with _ details). Raises: ValueError
– malformed config found.
-