ssb_timeseries.io.format

Definitions of the conventions an archive follows.

An archive is a file with a name, in a folder, written in a format. Which name, which folder and which format are all conventions, and conventions differ between organisations, so they are described here as data rather than built into the archiving code. See ssb_timeseries.io.archiving for the part that is not convention.

A convention is an ArchiveFormat. Several may be registered by name, and the archive is told which one to use. The generic definition is the fallback, and carries no assumptions about any particular organisation.

class ArchiveFormat(name, file_template, folder_order, data_format='parquet', version_pattern='_v(\\\\d+)$', version_prefix='_v', period_prefix='_p', period_template='{period_prefix}{from}{period_prefix}{to}', as_of_template=None, version_template='{version_prefix}{number}', timestamp_form='iso_no_colon', extension=None, allowed_characters=None, character_folds=<factory>, first_version=1, replicate_sharing=False)

Bases: object

How one organisation names and lays out its archives.

Every field is a convention rather than a mechanism. Two definitions that agree on all of them would produce identical archives for identical input, which is the only thing that distinguishes them.

Parameters:

  • name (str)

  • file_template (str)

  • folder_order (tuple[str, ...])

  • data_format (str)

  • version_pattern (str)

  • version_prefix (str)

  • period_prefix (str)

  • period_template (str)

  • as_of_template (str | None)

  • version_template (str)

  • timestamp_form (str)

  • extension (str | None)

  • allowed_characters (str | None)

  • character_folds (Mapping[str, str])

  • first_version (int)

  • replicate_sharing (bool)

allowed_characters: str | None = None

The characters a dataset name may be built from, or None for no restriction.

A name containing anything else is rejected, because a name that breaks the convention is worse than one that is refused.

as_of_template: str | None = None

The as-of segment, or None if a convention does not name the as-of.

Whether the as-of belongs in a file name at all is a convention, as is the shape it takes when it does, and both are settled here rather than in the template.

character_folds: Mapping[str, str]

Substitutions applied to a dataset name before it is checked or used.

Each key is a single character, and is replaced wherever it occurs, so a fold may lengthen the name as well as shorten it. These are recommended transliterations rather than replacements that carry meaning, so a name that folds to something else still names the same dataset.

data_format: str = 'parquet'

The format the data is written in, as understood by fs.write_dataframe().

extension: str | None = None

The file’s extension, if the convention requires a specific one.

None takes the extension implied by data_format.

property file_glob: str

The glob matching this convention’s archived files.

file_name(tokens, number)

Build the name of one archived file, without its extension.

Parameters:

  • tokens (dict[str, str]) – The naming tokens for the dataset.

  • number (int) – The version number to record in the name.

Return type:

str

Returns:

The file name.

file_template: str

The file name, without its extension, as a template over naming segments.

The template names whole segments rather than laying out characters, because the segments a convention wants are not all always wanted. A dataset with no period has no period to name, and a dataset with no as-of has no as-of marker, and a file name with a dangling separator in it says less than one without. The available segments are name, period, as_of and version, which are rendered from the conventions period_template, as_of_template and version_template below, and are each empty when the convention has nothing to say about them.

first_version: int = 1

The version number given to the first archive of a dataset.

folder_order: tuple[str, ...]

Which tokens become folders, outermost first, under the archive’s root.

This is a field of its own because it is the one part of a layout that a template cannot express: {product}/{process_stage} and {process_stage}/{product} are the same tokens in a different order. A token that is empty contributes no folder.

folder_path(tokens, root='')

Build the folder holding a dataset’s archives.

Parameters:

  • tokens (dict[str, str]) – The naming tokens, from which the folder tokens are taken.

  • root (str | PathLike[str]) – The archive’s root, which the folders are placed under.

Return type:

str | PathLike[str]

Returns:

The folder path, which may not yet exist.

name: str

The name this convention is selected by.

normalise(name)

Fold a dataset name into the characters this convention allows.

Parameters:

name (str) – The dataset name as it is known to this library.

Return type:

str

Returns:

The name to use in the archive’s paths.

Raises:

ValueError – If the name still contains disallowed characters after folding, since a silently rewritten name would be ambiguous.

period_prefix: str = '_p'

The prefix that introduces a period in a file name.

period_template: str = '{period_prefix}{from}{period_prefix}{to}'

The period segment, from the data’s own start and end.

Rendered only when the data has both, since a period with one end is not a period.

render_timestamp(value)

Render a date or datetime the way this convention does.

Parameters:

value (datetime | date) – The date or datetime to render.

Raises:

ValueError – If the convention names a renderer that does not exist.

Return type:

str

replicate_sharing: bool = False

Whether archiving also copies the archive to the dataset’s shared locations.

Some conventions require that shared data is archived wherever it is shared, others do not, so it is a property of the convention rather than a rule.

timestamp_form: str = 'iso_no_colon'

The name of the renderer used for the from, to and as_of tokens.

version_number(file_name)

Read the version number back out of an archived file’s name.

The extension is not part of the name a convention lays out, so it is taken off before matching.

Parameters:

file_name (str) – The name of an archived file.

Return type:

int | None

Returns:

The version number, or None if the name is not one of ours.

version_pattern: str = '_v(\\d+)$'

Regex matching the archive’s own version number in an existing file’s name.

Matched against the name without its extension, since a convention lays out the name and the format adds the extension. The capture group is the version number.

version_prefix: str = '_v'

The prefix that introduces a version number in a file name.

version_template: str = '{version_prefix}{number}'

The version segment, carrying the archive’s own version number.

FORMATS: dict[str, ArchiveFormat] = {'generic': ArchiveFormat(name='generic', file_template='{name}{version}', folder_order=('name',), data_format='parquet', version_pattern='_v(\\d+)$', version_prefix='_v', period_prefix='_p', period_template='{period_prefix}{from}{period_prefix}{to}', as_of_template=None, version_template='{version_prefix}{number}', timestamp_form='iso_no_colon', extension=None, allowed_characters=None, character_folds={}, first_version=1, replicate_sharing=False), 'ssb': ArchiveFormat(name='ssb', file_template='{name}{period}{as_of}{version}', folder_order=('product', 'process_stage', 'name'), data_format='parquet', version_pattern='_v(\\d+)$', version_prefix='_v', period_prefix='_p', period_template='{period_prefix}{from}{period_prefix}{to}', as_of_template='{version_prefix}{as_of}{version_prefix}', version_template='{version_prefix}{number}', timestamp_form='dashes_ms', extension=None, allowed_characters='abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789-_', character_folds={'æ': 'ae', 'ø': 'oe', 'å': 'aa', 'Æ': 'Ae', 'Ø': 'Oe', 'Å': 'Aa'}, first_version=1, replicate_sharing=True)}

The conventions this library knows, by name.

GENERIC = ArchiveFormat(name='generic', file_template='{name}{version}', folder_order=('name',), data_format='parquet', version_pattern='_v(\\d+)$', version_prefix='_v', period_prefix='_p', period_template='{period_prefix}{from}{period_prefix}{to}', as_of_template=None, version_template='{version_prefix}{number}', timestamp_form='iso_no_colon', extension=None, allowed_characters=None, character_folds={}, first_version=1, replicate_sharing=False)

The default convention, which assumes nothing about any organisation.

A dataset’s archives sit in a folder named after it, and its versions are numbered.

SSB = ArchiveFormat(name='ssb', file_template='{name}{period}{as_of}{version}', folder_order=('product', 'process_stage', 'name'), data_format='parquet', version_pattern='_v(\\d+)$', version_prefix='_v', period_prefix='_p', period_template='{period_prefix}{from}{period_prefix}{to}', as_of_template='{version_prefix}{as_of}{version_prefix}', version_template='{version_prefix}{number}', timestamp_form='dashes_ms', extension=None, allowed_characters='abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789-_', character_folds={'æ': 'ae', 'ø': 'oe', 'å': 'aa', 'Æ': 'Ae', 'Ø': 'Oe', 'Å': 'Aa'}, first_version=1, replicate_sharing=True)

The convention Statistics Norway’s archiving standards describe.

The folder levels are the product’s short name then the data state, as the navnestandard requires. The name is a short description, the data’s period, the as-of, and the version.

TIMESTAMP_FORMS: dict[str, Any] = {'dashes_ms': <function dashes_ms>, 'iso': <function iso>, 'iso_no_colon': <function iso_no_colon>}

Named renderers for a date or datetime inside a file name.

A convention names one of these rather than carrying a format string, so that the definitions stay declarative and each renderer can be tested on its own.

dashes_ms(dt)

Render a date or datetime with dashes in place of colons, in milliseconds.

The milliseconds are always written out, so that two file names differ only where their timestamps actually differ.

Return type:

str

Parameters:

dt (datetime | date)

get_format(name=None)

Look up a convention by name.

Parameters:

name (str | None) – The name of the convention, or None for the default.

Return type:

ArchiveFormat

Returns:

The convention.

Raises:

ValueError – If no convention by that name is known.

iso(dt)

Render a date or datetime in ISO 8601, keeping the colons.

Return type:

str

Parameters:

dt (datetime | date)

iso_no_colon(dt)

Render a date or datetime in ISO 8601, with the colons removed.

Return type:

str

Parameters:

dt (datetime | date)