kedro.config.TemplatedConfigLoader

class documentation

class TemplatedConfigLoader(AbstractConfigLoader): (source)

Extension of the ConfigLoader class that allows for template values, wrapped in brackets like: ${...}, to be automatically formatted based on the configs.

The easiest way to use this class is by setting the CONFIG_LOADER_CLASS constant in settings.py.

Example:

>>> # in settings.py
>>> from kedro.config import TemplatedConfigLoader
>>>
>>> CONFIG_LOADER_CLASS = TemplatedConfigLoader
>>> CONFIG_LOADER_ARGS = {
>>>     "globals_pattern": "*globals.yml",
>>> }

The contents of the dictionary resulting from the globals_pattern get merged with the globals_dict. In case of conflicts, the keys in globals_dict take precedence. If the formatting key is missing from the dictionary, the default template value is used (the format is "${key|default value}"). If no default is set, a ValueError will be raised.

Global parameters can be namespaced as well. An example could work as follows:

globals.yml

bucket: "my_s3_bucket"

environment: "dev"

datasets:
    csv: "pandas.CSVDataSet"
    spark: "spark.SparkDataSet"

folders:
    raw: "01_raw"
    int: "02_intermediate"
    pri: "03_primary"
    fea: "04_feature"

catalog.yml

raw_boat_data:
    type: "${datasets.spark}"
    filepath: "s3a://${bucket}/${environment}/${folders.raw}/boats.csv"
    file_format: parquet

raw_car_data:
    type: "${datasets.csv}"
    filepath: "s3://${bucket}/data/${environment}/${folders.raw}/cars.csv"

This uses jmespath in the background. For more information see: https://github.com/jmespath/jmespath.py and https://jmespath.org/.

Method	`__getitem__`	Undocumented
Method	`__init__`	Instantiates a `TemplatedConfigLoader`.
Method	`__repr__`	Undocumented
Method	`get`	Tries to resolve the template variables in the config dictionary provided by the `ConfigLoader` (super class) `get` method using the dictionary of replacement values obtained in the `__init__` method.
Instance Variable	`base_env`	Undocumented
Instance Variable	`config_patterns`	Undocumented
Instance Variable	`default_run_env`	Undocumented
Property	`conf_paths`	Property method to return deduplicated configuration paths.
Method	`_build_conf_paths`	Undocumented
Instance Variable	`_config_mapping`	Undocumented

Inherited from AbstractConfigLoader:

Instance Variable	`conf_source`	Undocumented
Instance Variable	`env`	Undocumented
Instance Variable	`runtime_params`	Undocumented

def __getitem__(self, key): (source) ¶

Undocumented

def __init__(self, conf_source: str, env: str = None, runtime_params: Dict[str, Any] = None, config_patterns: Dict[str, List[str]] = None, *, base_env: str = 'base', default_run_env: str = 'local', globals_pattern: Optional[str] = None, globals_dict: Optional[Dict[str, Any]] = None): (source) ¶

overrides kedro.config.AbstractConfigLoader.__init__

Instantiates a TemplatedConfigLoader.

Parameters
conf_source:`str`	Path to use as root directory for loading configuration.
env:`str`	Environment that will take precedence over base.
runtime_params:`Dict[str, Any]`	Extra parameters passed to a Kedro run.
config_patterns:`Dict[str, List[str]]`	Regex patterns that specify the naming convention for configuration files so they can be loaded. Can be customised by supplying config_patterns as in `CONFIG_LOADER_ARGS` in `settings.py`.
base_env:`str`
default_run_env:`str`
globals_pattern:`Optional[str]`	Optional keyword-only argument specifying a glob pattern. Files that match the pattern will be loaded as a formatting dictionary.
globals_dict:`Optional[Dict[str, Any]]`	Optional keyword-only argument specifying a formatting dictionary. This dictionary will get merged with the globals dictionary obtained from the globals_pattern. In case of duplicate keys, the `globals_dict` keys take precedence.

def __repr__(self): (source) ¶

Undocumented

def get(self, *patterns: str) -> Dict[str, Any]: (source) ¶

Tries to resolve the template variables in the config dictionary provided by the ConfigLoader (super class) get method using the dictionary of replacement values obtained in the __init__ method.

Parameters
*patterns:`str`	Glob patterns to match. Files, which names match any of the specified patterns, will be processed.
Returns
A Python dictionary with the combined configuration from all	configuration files. Note: any keys that start with `_` will be ignored. String values wrapped in `${...}` will be replaced with the result of the corresponding JMESpath expression evaluated against globals.
Raises
`ValueError`	malformed config found.

base_env = (source) ¶

Undocumented

config_patterns: dict = (source) ¶

Undocumented

default_run_env = (source) ¶

Undocumented

@property
conf_paths = (source) ¶

Property method to return deduplicated configuration paths.

def _build_conf_paths(self) -> Iterable[str]: (source) ¶

Undocumented

_config_mapping = (source) ¶

Undocumented