Pickle Storage

In addition to permanent and temporary storage, there is also short-term storage. Data can be stored here for 3-7 days. All data is stored in Pickle file format, meaning almost any Python object can be stored.

This storage is primarily used by the system, but it can also be used in modules.

Saving Data

Data is saved using the save_pickle function:

Arguments:

• path (pathlib.Path|str) - the file path.
  tip

  It is recommended to save files either in the /storage folder or in the module folder.

• data (Any) - the Python object to save.
• ttl (int) = Default value from config (DEFAULT_PICKLE_STORAGE_FILES_TTL) (= 259200 = 3 days) - The file's time-to-live in seconds. After this time, the file will be automatically deleted. It is recommended to set this to no more than 7 days (604800 seconds).

Returns: pathlib.Path - the full path to the file.

Example:

SomeModule/__init__.py

data = mod  # Save the entire module object, why not ¯\_(ツ)_/¯

storage_folder = mod.path / 'storage'
storage_folder.mkdir(parents=True, exist_ok=True)  # Create the folder if it doesn't exist
path = storage_folder / "module_obj.pkl"

# Save data without specifying TTL,  
# so the default value is used  
# (the user can change it in the config)
path = save_pickle(path, data)

mod.logger.info(f"Saved module object to {path}")

Loading Data

Data can be retrieved using load_pickle:

Arguments:

• path (pathlib.Path|str) - the file path.

Returns: Any - the data.

Example:

SomeModule/__init__.py

path = mod.path / 'storage' / 'module_obj.pkl'
loaded_mod = load_pickle(path)

Deleting Data

Simply delete the file, for example:

SomeModule/__init__.py

file_path = mod.path / 'storage' / 'module_obj.pkl'
if file_path.exists():
    file_path.unlink()

or:

SomeModule/__init__.py

import os

file_path = mod.path / 'storage' / 'module_obj.pkl'
if os.path.exists(file_path):
    os.remove(file_path)

Or just wait for the file to delete itself after the TTL expires 😁