kedro.io.AbstractDataSet

AbstractDataSet is the base class for all data set implementations. All data set implementations should extend this abstract class and implement the methods marked as abstract. If a specific dataset implementation cannot be used in conjunction with the ParallelRunner, such user-defined dataset should have the attribute _SINGLE_PROCESS = True. Example:

>>> from pathlib import Path, PurePosixPath
>>> import pandas as pd
>>> from kedro.io import AbstractDataSet
>>>
>>>
>>> class MyOwnDataSet(AbstractDataSet[pd.DataFrame, pd.DataFrame]):
>>>     def __init__(self, filepath, param1, param2=True):
>>>         self._filepath = PurePosixPath(filepath)
>>>         self._param1 = param1
>>>         self._param2 = param2
>>>
>>>     def _load(self) -> pd.DataFrame:
>>>         return pd.read_csv(self._filepath)
>>>
>>>     def _save(self, df: pd.DataFrame) -> None:
>>>         df.to_csv(str(self._filepath))
>>>
>>>     def _exists(self) -> bool:
>>>         return Path(self._filepath.as_posix()).exists()
>>>
>>>     def _describe(self):
>>>         return dict(param1=self._param1, param2=self._param2)

Example catalog.yml specification:

my_dataset:
    type: <path-to-my-own-dataset>.MyOwnDataSet
    filepath: data/01_raw/my_data.csv
    param1: <param1-value> # param1 is a required argument
    # param2 will be True by default