`ssb_timeseries.io.pyarrow_simple`¶

Provides a PyArrow-based simple, file-based I/O handler for Parquet format.

This handler stores datasets in a wide format (series as columns) with embedded metadata, using a defined directory structure. For example:

<repository_root>/
├── AS_OF_AT/
│   └── my_versioned_dataset/
│       ├── my_versioned_dataset-as_of_20230101T120000+0000-data.parquet
│       └── my_versioned_dataset-as_of_20230102T120000+0000-data.parquet
└── NONE_AT/
    └── my_dataset/
        └── my_dataset-latest-data.parquet

It uses PyArrow for eager reading and writing.

class FileSystem(repository, path='', **options)¶

Bases: object

A filesystem abstraction for reading and writing dataset data.

Parameters:

repository (Any)
path (str)
options (Any)

__init__(repository, path='', **options)¶

Initialize the filesystem handler for a repository.

Which dataset an operation concerns is passed to that operation as a DatasetRef, so a single handler can serve any number of datasets.

Parameters:

repository (typing.Any) – The repository configuration dictionary, or the name of a repository in the active configuration.
path (str) – Root path of the repository, passed in from the repository binding’s options by the dispatcher.
**options (typing.Any) – Handler specific options. Unknown keys are accepted and retained so that handlers can be extended without changing the dispatcher.

Return type:

None

exists(dataset)¶

Check if the data file for a dataset exists.

Parameters:: dataset (DatasetRef) – The dataset to check.
Return type:: bool

fullpath(dataset)¶

Return the full path to a dataset’s data file.

Return type:: str
Parameters:: dataset (DatasetRef)

read(dataset, interval='')¶

Read a dataset’s data from the filesystem.

Returns an empty dataframe if the file is not found.

Parameters:

dataset (DatasetRef) – The dataset to read.
interval (str) – The part of the dataset to read. Not yet implemented.

Return type:

Table

property root: str¶: Return the root path of the configured repository.

versions(dataset, file_pattern=None)¶

List all available version markers from a dataset’s data directory.

The versioning comes from the dataset ref, so no pattern argument is needed to know how this library names its versioned files.

Parameters:

dataset (DatasetRef) – The dataset to list versions for.
file_pattern (str | None) – Glob selecting the files to inspect, which is a property of this handler’s storage rather than of the dataset. The default, None, reads it from the repository options and falls back to "*.parquet", the only suffix this handler writes.

Return type:

list[datetime | str]

write(dataset, data, tags=None)¶

Write a dataset’s data to the filesystem.

If versioning is AS_OF, a new file is always created. If versioning is NONE, new data is merged into the existing file.

Parameters:

dataset (DatasetRef) – The dataset to write.
data (narwhals.typing.FrameT) – The data to write.
tags (dict | None) – The dataset’s tags, used to derive the storage schema.

Return type:

None

ssb_timeseries.io.pyarrow_simple¶

`ssb_timeseries.io.pyarrow_simple`¶