kedro.config.OmegaConfigLoader

class documentation

class OmegaConfigLoader(AbstractConfigLoader): (source)

Recursively scan directories (config paths) contained in conf_source for configuration files with a yaml, yml or json extension, load and merge them through OmegaConf (https://omegaconf.readthedocs.io/) and return them in the form of a config dictionary.

The first processed config path is the base directory inside conf_source. The optional env argument can be used to specify a subdirectory of conf_source to process as a config path after base.

When the same top-level key appears in any two config files located in the same (sub)directory, a ValueError is raised.

When the same key appears in any two config files located in different (sub)directories, the last processed config path takes precedence and overrides this key and any sub-keys.

You can access the different configurations as follows:

>>> import logging.config
>>> from kedro.config import OmegaConfigLoader
>>> from kedro.framework.project import settings
>>>
>>> conf_path = str(project_path / settings.CONF_SOURCE)
>>> conf_loader = OmegaConfigLoader(conf_source=conf_path, env="local")
>>>
>>> conf_logging = conf_loader["logging"]
>>> logging.config.dictConfig(conf_logging)  # set logging conf
>>>
>>> conf_catalog = conf_loader["catalog"]
>>> conf_params = conf_loader["parameters"]

OmegaConf supports variable interpolation in configuration https://omegaconf.readthedocs.io/en/2.2_branch/usage.html#merging-configurations. It is recommended to use this instead of yaml anchors with the OmegaConfigLoader.

This version of the OmegaConfigLoader does not support any of the built-in OmegaConf resolvers. Support for resolvers might be added in future versions.

To use this class, change the setting for the CONFIG_LOADER_CLASS constant in settings.py.

Example:

>>> # in settings.py
>>> from kedro.config import OmegaConfigLoader
>>>
>>> CONFIG_LOADER_CLASS = OmegaConfigLoader

Method	`__getitem__`	Get configuration files by key, load and merge them, and return them in the form of a config dictionary.
Method	`__init__`	Instantiates a `OmegaConfigLoader`.
Method	`__repr__`	Undocumented
Method	`load_and_merge_dir_config`	Recursively load and merge all configuration files in a directory using OmegaConf, which satisfy a given list of glob patterns from a specific path.
Instance Variable	`base_env`	Undocumented
Instance Variable	`config_patterns`	Undocumented
Instance Variable	`default_run_env`	Undocumented
Static Method	`_check_duplicates`	Undocumented
Static Method	`_clear_omegaconf_resolvers`	Clear the built-in OmegaConf resolvers.
Static Method	`_is_valid_config_path`	Check if given path is a file path and file type is yaml or json.
Static Method	`_resolve_environment_variables`	Use the `oc.env` resolver to read environment variables and replace them in-place, clearing the resolver after the operation is complete if it was not registered beforehand.

Inherited from AbstractConfigLoader:

Instance Variable	`conf_source`	Undocumented
Instance Variable	`env`	Undocumented
Instance Variable	`runtime_params`	Undocumented

def __getitem__(self, key) -> Dict[str, Any]: (source) ¶

Get configuration files by key, load and merge them, and return them in the form of a config dictionary.

Parameters
key	Key of the configuration type to fetch.
Returns
`Dict[str`, `Any]`	A Python dictionary with the combined configuration from all configuration files.
Raises
`KeyError`	If key provided isn't present in the config_patterns of this `OmegaConfigLoader` instance.
`MissingConfigException`	If no configuration files exist matching the patterns mapped to the provided key.

def __init__(self, conf_source: str, env: str = None, runtime_params: Dict[str, Any] = None, *, config_patterns: Dict[str, List[str]] = None, base_env: str = 'base', default_run_env: str = 'local'): (source) ¶

overrides kedro.config.AbstractConfigLoader.__init__

Instantiates a OmegaConfigLoader.

Parameters
conf_source:`str`	Path to use as root directory for loading configuration.
env:`str`	Environment that will take precedence over base.
runtime_params:`Dict[str, Any]`	Extra parameters passed to a Kedro run.
config_patterns:`Dict[str, List[str]]`	Regex patterns that specify the naming convention for configuration files so they can be loaded. Can be customised by supplying config_patterns as in `CONFIG_LOADER_ARGS` in `settings.py`.
base_env:`str`	Name of the base environment. Defaults to `"base"`. This is used in the `conf_paths` property method to construct the configuration paths.
default_run_env:`str`	Name of the default run environment. Defaults to `"local"`. Can be overridden by supplying the `env` argument.

def __repr__(self): (source) ¶

Undocumented

def load_and_merge_dir_config(self, conf_path: str, patterns: Iterable[str], read_environment_variables: Optional[bool] = False) -> Dict[str, Any]: (source) ¶

Recursively load and merge all configuration files in a directory using OmegaConf, which satisfy a given list of glob patterns from a specific path.

Parameters
conf_path:`str`	Path to configuration directory.
patterns:`Iterable[str]`	List of glob patterns to match the filenames against.
read_environment_variables:`Optional[bool]`	Whether to resolve environment variables.
Returns
`Dict[str, Any]`	Resulting configuration dictionary.
Raises
`MissingConfigException`	If configuration path doesn't exist or isn't valid.
`ValueError`	If two or more configuration files contain the same key(s).
`ParserError`	If config file contains invalid YAML or JSON syntax.

base_env = (source) ¶

Undocumented

config_patterns: dict = (source) ¶

Undocumented

default_run_env = (source) ¶

Undocumented

@staticmethod
def _check_duplicates(seen_files_to_keys: Dict[Path, Set[Any]]): (source) ¶

Undocumented

@staticmethod
def _clear_omegaconf_resolvers(): (source) ¶

Clear the built-in OmegaConf resolvers.

@staticmethod
def _is_valid_config_path(path): (source) ¶

Check if given path is a file path and file type is yaml or json.

@staticmethod
def _resolve_environment_variables(config: Dict[str, Any]): (source) ¶

Use the oc.env resolver to read environment variables and replace them in-place, clearing the resolver after the operation is complete if it was not registered beforehand.

Parameters
config:`Dict[str, Any]`	Undocumented
config {Dict[str
Any]} -- The configuration dictionary to resolve.