kedro.pipeline.modular_pipeline

module documentation
(source)

Helper to integrate modular pipelines into a master pipeline.

Exception ModularPipelineError Raised when a modular pipeline is not adapted and integrated appropriately using the helper.
Function _get_dataset_names_mapping Take a name or a collection of dataset names and turn it into a mapping from the old dataset names to the provided ones if necessary.
Function _get_param_names_mapping Take a parameter or a collection of parameter names and turn it into a mapping from existing parameter names to new ones if necessary. It follows the same rule as _get_dataset_names_mapping and prefixes the keys on the resultant dictionary with ...
Function _is_all_parameters Undocumented
Function _is_parameter Undocumented
Function _is_single_parameter Undocumented
Function _normalize_param_name Make sure that a param name has a params: prefix before passing to the node
Function _validate_datasets_exist Undocumented
Function _validate_inputs_outputs Safeguards to ensure that: - parameters are not specified under inputs - inputs are only free inputs - outputs do not contain free inputs
def _get_dataset_names_mapping(names: Union[str, Set[str], Dict[str, str]] = None) -> Dict[str, str]: (source)

Take a name or a collection of dataset names and turn it into a mapping from the old dataset names to the provided ones if necessary.

Examples

>>> _get_dataset_names_mapping("dataset_name")
{"dataset_name": "dataset_name"}  # a str name will stay the same
>>> _get_dataset_names_mapping(set(["ds_1", "ds_2"]))
{"ds_1": "ds_1", "ds_2": "ds_2"}  # a Set[str] of names will stay the same
>>> _get_dataset_names_mapping({"ds_1": "new_ds_1_name"})
{"ds_1": "new_ds_1_name"}  # a Dict[str, str] of names will map key to value
Parameters
names:Union[str, Set[str], Dict[str, str]]A dataset name or collection of dataset names. When str or Set[str] is provided, the listed names will stay the same as they are named in the provided pipeline. When Dict[str, str] is provided, current names will be mapped to new names in the resultant pipeline.
Returns
Dict[str, str]A dictionary that maps the old dataset names to the provided ones.
def _get_param_names_mapping(names: Union[str, Set[str], Dict[str, str]] = None) -> Dict[str, str]: (source)

Take a parameter or a collection of parameter names and turn it into a mapping from existing parameter names to new ones if necessary. It follows the same rule as _get_dataset_names_mapping and prefixes the keys on the resultant dictionary with params: to comply with node's syntax.

Examples

>>> _get_param_names_mapping("param_name")
{"params:param_name": "params:param_name"}  # a str name will stay the same
>>> _get_param_names_mapping(set(["param_1", "param_2"]))
# a Set[str] of names will stay the same
{"params:param_1": "params:param_1", "params:param_2": "params:param_2"}
>>> _get_param_names_mapping({"param_1": "new_name_for_param_1"})
# a Dict[str, str] of names will map key to valu
{"params:param_1": "params:new_name_for_param_1"}
Parameters
names:Union[str, Set[str], Dict[str, str]]A parameter name or collection of parameter names. When str or Set[str] is provided, the listed names will stay the same as they are named in the provided pipeline. When Dict[str, str] is provided, current names will be mapped to new names in the resultant pipeline.
Returns
Dict[str, str]A dictionary that maps the old parameter names to the provided ones.
def _is_all_parameters(name: str) -> bool: (source)

Undocumented

def _is_parameter(name: str) -> bool: (source)

Undocumented

def _is_single_parameter(name: str) -> bool: (source)

Undocumented

def _normalize_param_name(name: str) -> str: (source)

Make sure that a param name has a params: prefix before passing to the node

def _validate_datasets_exist(inputs: AbstractSet[str], outputs: AbstractSet[str], parameters: AbstractSet[str], pipe: Pipeline): (source)

Undocumented

def _validate_inputs_outputs(inputs: AbstractSet[str], outputs: AbstractSet[str], pipe: Pipeline): (source)

Safeguards to ensure that: - parameters are not specified under inputs - inputs are only free inputs - outputs do not contain free inputs