ssb_timeseries.io.fs

The main purpose of the ssb_timeseries.fs module is to allow file based IO that works on both a local file system and Google Cloud Storage.

cp(from_path, to_path)

Copy file from one location to another.

This function handles copying files between local and GCS paths, automatically selecting the correct backend.

Parameters:

  • from_path (str | PathLike[str]) – The path to the source file.

  • to_path (str | PathLike[str]) – The path to the destination file.

Return type:

None

existing_subpath(path)

Return the existing part of a path on local or GCS file system.

Return type:

str | PathLike[str]

Parameters:

path (str | PathLike[str])

exists(path)

Check if a given (local or GCS) path exists.

Return type:

bool

Parameters:

path (str | PathLike[str])

file_count(path, create=False)

Count files in path. Should work regardless of wether source and target location is local fs or GCS to local.

Return type:

int

Parameters:

  • path (str | PathLike[str])

  • create (bool)

find(search_path, equals='', contains='', pattern='', search_sub_dirs=True, replace_root=False, recursive=False)

Find files and subdirectories under a local or GCS path.

At most one of equals, contains and pattern may be given; they are alternatives, not independent filters. With none of them, the name criterion is *, so everything at the searched depth matches.

The searched depth is decided by recursive and search_sub_dirs, which build these search patterns against search_path:

  • neither: search_path/pattern, so search_path itself.

  • search_sub_dirs, the default: search_path/*/pattern, one level below.

  • recursive: search_path/**/pattern, every depth below, including search_path itself.

Files and subdirectories both match, and the result is sorted.

Parameters:

  • search_path (str | PathLike[str]) – Root to search, a local path or a gs://<bucket>/… URL.

  • equals (str) – Name that must match exactly.

  • contains (str) – Substring the name must include, matched as *contains*.

  • pattern (str) – Glob pattern for the name, for example *.json.

  • search_sub_dirs (bool) – Search the immediate subdirectories of search_path instead of search_path itself.

  • replace_root (bool) – Rewrite search_path at the start of each returned path as the literal root, for example root/sub1/c.json. The rewritten paths always use /, on every platform.

  • recursive (bool) – Search every depth below search_path, overriding search_sub_dirs.

Return type:

list[str]

Returns:

Sorted list of full paths of the matches. Equally named entries in different subdirectories stay distinct, because paths are never reduced to their final component. On GCS the paths carry no gs://<bucket> prefix.

Raises:

ValueError – If more than one of equals, contains and pattern is given.

fs_type(path)

Check filesystem type (local or GCS) for a given path.

Return type:

str

Parameters:

path (str | PathLike[str])

is_gcs(path)

Check if path is on GCS.

Return type:

bool

Parameters:

path (str | PathLike[str])

is_local(path)

Check if path is local.

Return type:

bool

Parameters:

path (str | PathLike[str])

ls(path, pattern='*', create=False)

List files. Should work regardless of wether the filesystem is local or GCS.

Return type:

list[str]

Parameters:

  • path (str)

  • pattern (str)

  • create (bool)

mk_parent_dir(path)

Ensure a parent directory exists. … regardless of wether fielsystem is local or GCS.

Return type:

None

Parameters:

path (str | PathLike[str])

mkdir(path)

Make directory regardless of filesystem is local or GCS.

Return type:

None

Parameters:

path (str | PathLike[str])

mv(from_path, to_path)

Move file from one location to another.

This function handles moving files between local and GCS paths, automatically selecting the correct backend.

Parameters:

  • from_path (str | PathLike[str]) – The path to the source file.

  • to_path (str | PathLike[str]) – The path to the destination file.

Return type:

None

path(*args)

Join args to form path. Make sure that gcs paths are begins with double slash: gs://…

Return type:

str

Parameters:

args (str | PathLike[str])

path_to_str(path)

Normalise as strings.

This is a trick to make automated tests pass on Windows.

Return type:

str | PathLike[str]

Parameters:

path (str | PathLike[str])

read_json(path)

Read json file from path on either local fs or GCS.

Return type:

dict

Parameters:

path (str | PathLike[str])

read_parquet(path, lazy=False, implementation='pyarrow', **kwargs)

Read a Parquet file into a dataframe.

This function can read from both local and GCS paths.

Parameters:

  • path (str | PathLike[str]) – The path to the Parquet file.

  • lazy (bool) – If True, returns a lazy dataframe. Defaults to False.

  • implementation (str) – The backend to use for reading the file. Defaults to “pyarrow”.

  • **kwargs – Additional keyword arguments passed to the backend.

Return type:

Union[narwhals.DataFrame[typing.Any], narwhals.LazyFrame[typing.Any]]

Returns:

A Narwhals dataframe.

read_text(path, file_format='')

Read a text file from specified path on either local fs or GCS.

Return type:

dict

Parameters:

  • path (str | PathLike[str])

  • file_format (str)

remove_prefix(path)

Helper function to compensate for some os.* functions shorten gs://<path> to gs:/<path>.

Return type:

str

Parameters:

path (str | PathLike[str])

rm(path)

Remove a file from either the local filesystem or GCS.

This function is non-recursive. For a recursive variant, see rmtree().

Parameters:

path (str | PathLike[str]) – The path to the file to be removed.

Return type:

None

rmtree(path)

Recursively remove a directory and all its subdirectories and files regardless of local or GCS filesystem.

Return type:

None

Parameters:

path (str)

same_path(*args)

Return common part of path, for two or more files. Files must be on same file system, but the file system can be either local or GCS.

Return type:

str | PathLike[str]

touch(path)

Touch file regardless of wether the filesystem is local or GCS; return path.

Return type:

str | PathLike[str]

Parameters:

path (str | PathLike[str])

wrap_return_as_str(func)

Decorator to normalise outputs using path_to_str().

Return type:

Callable

Parameters:

func (Callable)

write_dataframe(data, path, file_format, **kwargs)

Write a dataframe to a file in the named format.

The counterpart of write_parquet() for any other format, so that choosing a format is a matter of naming it rather than of calling a different function. Works on both local and GCS paths, selecting the backend automatically.

Parameters:

  • data (Table | NativeDataFrame | NativeLazyFrame | NativeIbis) – The dataframe to write (can be a PyArrow Table or any Narwhals-compatible dataframe).

  • path (str | PathLike[str]) – The destination path for the file.

  • file_format (str) – The name of the format, for example “parquet” or “csv”.

  • **kwargs – Additional keyword arguments passed to the backend.

Raises:

ValueError – If the format is not supported.

Return type:

None

write_json(path, content)

Write json file to path on either local fs or GCS.

Return type:

None

Parameters:

  • path (str | PathLike[str])

  • content (str | dict)

write_parquet(data, path, schema=None, **kwargs)

Write a dataframe to a Parquet file.

This function can write to both local and GCS paths, automatically selecting the correct filesystem backend. It also handles schema validation.

Parameters:

  • data (Table | NativeDataFrame | NativeLazyFrame | NativeIbis) – The dataframe to write (can be a PyArrow Table or any Narwhals-compatible dataframe).

  • path (str | PathLike[str]) – The destination path for the Parquet file.

  • schema (Schema | None) – An optional PyArrow schema to validate against before writing.

  • **kwargs – Additional keyword arguments passed to the backend.

Return type:

None

write_text(path, content, file_format)

Write json file to path on either local fs or GCS.

Return type:

None

Parameters:

  • path (str | PathLike[str])

  • content (str | dict)

  • file_format (str)