kedro.io.IncrementalDataSet

IncrementalDataSet inherits from PartitionedDataSet, which loads and saves partitioned file-like data using the underlying dataset definition. For filesystem level operations it uses fsspec: https://github.com/intake/filesystem_spec. IncrementalDataSet also stores the information about the last processed partition in so-called checkpoint that is persisted to the location of the data partitions by default, so that subsequent pipeline run loads only new partitions past the checkpoint.

Example:

>>> from kedro.io import IncrementalDataSet
>>>
>>> # these credentials will be passed to:
>>> # a) 'fsspec.filesystem()' call,
>>> # b) the dataset initializer,
>>> # c) the checkpoint initializer
>>> credentials = {"key1": "secret1", "key2": "secret2"}
>>>
>>> data_set = IncrementalDataSet(
>>>     path="s3://bucket-name/path/to/folder",
>>>     dataset="pandas.CSVDataSet",
>>>     credentials=credentials
>>> )
>>> loaded = data_set.load()  # loads all available partitions
>>> # assert isinstance(loaded, dict)
>>>
>>> data_set.confirm()  # update checkpoint value to the last processed partition ID
>>> reloaded = data_set.load()  # still loads all available partitions
>>>
>>> data_set.release()  # clears load cache
>>> # returns an empty dictionary as no new partitions were added
>>> data_set.load()