kedro.runner.parallel

module documentation

(source)

ParallelRunner is an AbstractRunner implementation. It can be used to run the Pipeline in parallel groups formed by toposort.

Class	`ParallelRunnerManager`	`ParallelRunnerManager` is used to create shared `MemoryDataSet` objects as default data sets in a pipeline.
Class	`_SharedMemoryDataSet`	`_SharedMemoryDataSet` is a wrapper class for a shared MemoryDataSet in SyncManager. It is not inherited from AbstractDataSet class.
Function	`_bootstrap_subprocess`	Undocumented
Function	`_run_node_synchronization`	Run a single `Node` with inputs from and outputs to the `catalog`.
Constant	`_MAX_WINDOWS_WORKERS`	Undocumented

def _bootstrap_subprocess(package_name: str, logging_config: Dict[str, Any]): (source) ¶

Undocumented

def _run_node_synchronization(node: Node, catalog: DataCatalog, is_async: bool = False, session_id: str = None, package_name: str = None, logging_config: Dict[str, Any] = None) -> Node: (source) ¶

Run a single Node with inputs from and outputs to the catalog.

A PluginManager instance is created in each subprocess because the PluginManager can't be serialised.

Parameters
node:`Node`	The `Node` to run.
catalog:`DataCatalog`	A `DataCatalog` containing the node's inputs and outputs.
is_async:`bool`	If True, the node inputs and outputs are loaded and saved asynchronously with threads. Defaults to False.
session_id:`str`	The session id of the pipeline run.
package_name:`str`	The name of the project Python package.
logging_config:`Dict[str, Any]`	A dictionary containing logging configuration.
Returns
`Node`	The node argument.

Undocumented

Value

kedro.runner.parallel_runner