kedro.extras.datasets.pickle.PickleDataSet

PickleDataSet loads/saves data from/to a Pickle file using an underlying filesystem (e.g.: local, S3, GCS). The underlying functionality is supported by the specified backend library passed in (defaults to the pickle library), so it supports all allowed options for loading and saving pickle files.

Example usage for the YAML API:

test_model: # simple example without compression
  type: pickle.PickleDataSet
  filepath: data/07_model_output/test_model.pkl
  backend: pickle

final_model: # example with load and save args
  type: pickle.PickleDataSet
  filepath: s3://your_bucket/final_model.pkl.lz4
  backend: joblib
  credentials: s3_credentials
  save_args:
    compress: lz4

Example usage for the Python API:

>>> from kedro.extras.datasets.pickle import PickleDataSet
>>> import pandas as pd
>>>
>>> data = pd.DataFrame({'col1': [1, 2], 'col2': [4, 5],
>>>                      'col3': [5, 6]})
>>>
>>> data_set = PickleDataSet(filepath="test.pkl", backend="pickle")
>>> data_set.save(data)
>>> reloaded = data_set.load()
>>> assert data.equals(reloaded)
>>>
>>> data_set = PickleDataSet(filepath="test.pickle.lz4",
>>>                          backend="compress_pickle",
>>>                          load_args={"compression":"lz4"},
>>>                          save_args={"compression":"lz4"})
>>> data_set.save(data)
>>> reloaded = data_set.load()
>>> assert data.equals(reloaded)