kedro.extras.datasets.redis.PickleDataSet

class documentation

class PickleDataSet(AbstractDataSet[Any, Any]): (source)

View In Hierarchy

PickleDataSet loads/saves data from/to a Redis database. The underlying functionality is supported by the redis library, so it supports all allowed options for instantiating the redis app from_url and setting a value.

Example usage for the YAML API:

my_python_object: # simple example
  type: redis.PickleDataSet
  key: my_object
  from_url_args:
    url: redis://127.0.0.1:6379

final_python_object: # example with save args
  type: redis.PickleDataSet
  key: my_final_object
  from_url_args:
    url: redis://127.0.0.1:6379
    db: 1
  save_args:
    ex: 10

Example usage for the Python API:

>>> from kedro.extras.datasets.redis import PickleDataSet
>>> import pandas as pd
>>>
>>> data = pd.DataFrame({'col1': [1, 2], 'col2': [4, 5],
>>>                       'col3': [5, 6]})
>>>
>>> my_data = PickleDataSet(key="my_data")
>>> my_data.save(data)
>>> reloaded = my_data.load()
>>> assert data.equals(reloaded)
Method __init__ Creates a new instance of PickleDataSet. This loads/saves data from/to a Redis database while deserialising/serialising. Supports custom backends to serialise/deserialise objects.
Constant DEFAULT_LOAD_ARGS Undocumented
Constant DEFAULT_REDIS_URL Undocumented
Constant DEFAULT_SAVE_ARGS Undocumented
Method _describe Undocumented
Method _exists Undocumented
Method _load Undocumented
Method _save Undocumented
Instance Variable _backend Undocumented
Instance Variable _key Undocumented
Instance Variable _load_args Undocumented
Instance Variable _redis_db Undocumented
Instance Variable _redis_from_url_args Undocumented
Instance Variable _redis_set_args Undocumented
Instance Variable _save_args Undocumented

Inherited from AbstractDataSet:

Class Method from_config Create a data set instance using the configuration provided.
Method __str__ Undocumented
Method exists Checks whether a data set's output already exists by calling the provided _exists() method.
Method load Loads data by delegation to the provided load method.
Method release Release any cached data.
Method save Saves data by delegation to the provided save method.
Method _copy Undocumented
Method _release Undocumented
Property _logger Undocumented
def __init__(self, key: str, backend: str = 'pickle', load_args: Dict[str, Any] = None, save_args: Dict[str, Any] = None, credentials: Dict[str, Any] = None, redis_args: Dict[str, Any] = None): (source)

Creates a new instance of PickleDataSet. This loads/saves data from/to a Redis database while deserialising/serialising. Supports custom backends to serialise/deserialise objects.

Example backends that are compatible (non-exhaustive):
Example backends that are incompatible:
  • torch
Parameters
key:strThe key to use for saving/loading object to Redis.
backend:strBackend to use, must be an import path to a module which satisfies the pickle interface. That is, contains a loads and dumps function. Defaults to 'pickle'.
load_args:Dict[str, Any]Pickle options for loading pickle files. You can pass in arguments that the backend load function specified accepts, e.g: pickle.loads: https://docs.python.org/3/library/pickle.html#pickle.loads dill.loads: https://dill.readthedocs.io/en/latest/index.html#dill.loads compress_pickle.loads: https://lucianopaz.github.io/compress_pickle/html/api/compress_pickle.html#compress_pickle.compress_pickle.loads All defaults are preserved.
save_args:Dict[str, Any]Pickle options for saving pickle files. You can pass in arguments that the backend dump function specified accepts, e.g: pickle.dumps: https://docs.python.org/3/library/pickle.html#pickle.dump dill.dumps: https://dill.readthedocs.io/en/latest/index.html#dill.dumps compress_pickle.dumps: https://lucianopaz.github.io/compress_pickle/html/api/compress_pickle.html#compress_pickle.compress_pickle.dumps All defaults are preserved.
credentials:Dict[str, Any]Credentials required to get access to the redis server. E.g. {"password": None}.
redis_args:Dict[str, Any]Extra arguments to pass into the redis client constructor redis.StrictRedis.from_url. (e.g. {"socket_timeout": 10}), as well as to pass to the redis.StrictRedis.set through nested keys from_url_args and set_args. Here you can find all available arguments for from_url: https://redis-py.readthedocs.io/en/stable/connections.html?highlight=from_url#redis.Redis.from_url All defaults are preserved, except url, which is set to redis://127.0.0.1:6379. You could also specify the url through the env variable REDIS_URL.
Raises
ValueErrorIf backend does not satisfy the pickle interface.
ImportErrorIf the backend module could not be imported.
DEFAULT_LOAD_ARGS: Dict[str, Any] = (source)

Undocumented

Value
{}
DEFAULT_REDIS_URL = (source)

Undocumented

Value
os.getenv('REDIS_URL', 'redis://127.0.0.1:6379')
DEFAULT_SAVE_ARGS: Dict[str, Any] = (source)

Undocumented

Value
{}
def _describe(self) -> Dict[str, Any]: (source)
overrides kedro.io.AbstractDataSet._describe

Undocumented

def _exists(self) -> bool: (source)
overrides kedro.io.AbstractDataSet._exists

Undocumented

def _load(self) -> Any: (source)
overrides kedro.io.AbstractDataSet._load

Undocumented

def _save(self, data: Any): (source)
overrides kedro.io.AbstractDataSet._save

Undocumented

_backend = (source)

Undocumented

_key = (source)

Undocumented

_load_args = (source)

Undocumented

_redis_db = (source)

Undocumented

_redis_from_url_args = (source)

Undocumented

_redis_set_args = (source)

Undocumented

_save_args = (source)

Undocumented