ssb_timeseries.io.pyarrow_hive

Provides a Hive-partitioned, file-based I/O handler for Parquet format.

This handler stores datasets in a wide format (series as columns) with embedded metadata, using a defined Hive-partitioned directory structure. For example:

<repository_root>/
├── data_type=AS_OF_AT/
│   └── dataset=my_versioned_dataset/
│       ├── as_of=2023-01-01T120000+0000/
│       │   └── part-0.parquet
│       └── as_of=2023-01-02T120000+0000/
│           └── part-0.parquet
└── data_type=NONE_AT/
    └── dataset=my_dataset/
        └── as_of=__HIVE_DEFAULT_PARTITION__/
            └── part-0.parquet
class HiveFileSystem(repository, path='', **kwargs)

Bases: object

A filesystem abstraction for reading and writing Hive-partitioned datasets.

Parameters:

  • repository (Any)

  • path (str)

  • kwargs (dict[str, Any])

__init__(repository, path='', **kwargs)

Initialize the filesystem handler for a repository.

Which dataset an operation concerns is passed to that operation as a DatasetRef, so a single handler can serve any number of datasets.

Parameters:

  • repository (typing.Any) – The repository configuration dictionary, or the name of a repository in the active configuration.

  • path (str) – Root path of the repository, passed in from the repository binding’s options by the dispatcher.

  • **kwargs (dict[str, typing.Any]) – Handler specific options. Unknown keys are accepted and retained so that handlers can be extended without changing the dispatcher.

Return type:

None

exists(dataset)

Check if a dataset’s directory exists.

Parameters:

dataset (DatasetRef) – The dataset to check.

Return type:

bool

read(dataset, *args, **kwargs)

Read a partitioned dataset from the filesystem.

Parameters:

  • dataset (DatasetRef) – The dataset to read.

  • *args – Accepted and ignored, for call compatibility.

  • **kwargs – Accepted and ignored, for call compatibility.

Return type:

narwhals.typing.FrameT

property root: str

Return the root path of the configured repository.

versions(dataset)

List available versions by inspecting a dataset’s subdirectories.

Hive keeps its data in partition directories rather than in versioned file names, so there is no file name pattern to configure and no filename convention to parse. The versioning comes from the ref.

Parameters:

dataset (DatasetRef) – The dataset to list versions for.

Return type:

list[datetime | str]

write(dataset, data, tags=None)

Write a dataset’s data to the filesystem, partitioned by versioning scheme.

Parameters:

  • dataset (DatasetRef) – The dataset to write.

  • data (narwhals.typing.FrameT) – The data to write.

  • tags (dict | None) – The dataset’s tags, used to derive the storage schema.

Return type:

None