dapla_metadata.standards package

Subpackages

dapla_metadata.standards.dataset_path module

Build complete or partial paths from validated dataset metadata.

class DataState(*values)[source]

Bases: StrEnum

Data states supported by the dataset path generator.

SOURCE_DATA is excluded: kildedata filenames are not covered by the naming standard that paths are generated from.

INPUT_DATA = 'inndata'
OUTPUT_DATA = 'utdata'
PROCESSED_DATA = 'klargjorte-data'
STATISTICS = 'statistikk'
class FileType(*values)[source]

Bases: StrEnum

File types supported by the dataset path generator.

CSV = 'csv'
JSON = 'json'
PARQUET = 'parquet'
XML = 'xml'
create_dataset_path(*, bucket=None, product=None, data_state=None, short_description=None, period_from=None, period_to=None, version=None, file_type=None, folders=None)[source]

Build a complete or contiguous partial path from dataset metadata.

The complete path has the form:

gs://{bucket}/{product}/{data_state}/{folders...}/{short_description}_p{period_from}[_p{period_to}]_v{version}.{file_type}

Any argument can be combined as long as they follow the correct order.

A filename is atomic: short_description, period_from, version, and file_type must either all be provided or all be omitted. period_to is optional, but can only be used as part of a complete filename.

Callers should provide semantic values only. The function should add the gs://, _p, _v, and . syntax itself. It should not access GCS, inspect the filesystem, or silently correct invalid input.

Parameters:

  • bucket (str | None) – Optional bucket name, without the gs:// prefix. It must be a non-empty single path segment: no slashes and no leading or trailing whitespace. Beyond that the name is included verbatim and is not validated against the full GCS naming rules, so callers are responsible for validating it when necessary. For example, "ssb-dapla-example-data-produkt-prod".

  • product (str | None) – Optional non-empty statistics-product or data-product short name. It may contain uppercase and lowercase letters, digits, -, and _. For example, "ledstill" or "ameld_data".

  • data_state (DataState | None) – An optional DataState enum member: DataState.INPUT_DATA, DataState.PROCESSED_DATA, DataState.STATISTICS, or DataState.OUTPUT_DATA. Do not pass a raw string.

  • short_description (str | None) – Optional non-empty dataset short description. It must contain only letters, digits, and hyphens, with no restriction on hyphen placement. Underscores, spaces, slashes, and periods are not accepted. For example, "varehandel", "grensehandel-imputert", or "-imputert--data-".

  • period_from (str | None) –

    The optional first period as a string. Do not include the p prefix. Supported formats are:

    • year: YYYY — for example, "2019";

    • month: YYYY-MM — for example, "2022-10";

    • calendar date: YYYY-MM-DD — for example, "2022-01-24";

    • ISO week: YYYY-Www — for example, "2020-W15";

    • ISO ordinal date: YYYY-DDD — for example, "2022-015";

    • date and time: YYYY-MM-DDTHH-MM-SS.sss — for example, "2024-12-31T23-59-30.000";

    • SSB bimonthly period: YYYY-Bn, where n is 1-6;

    • SSB quarterly period: YYYY-Qn, where n is 1-4;

    • SSB four-month period: YYYY-Tn, where n is 1-3;

    • SSB half-year period: YYYY-Hn, where n is 1-2.

  • period_to (str | None) – An optional second period as a string, without the p prefix. If supplied, it must use the same format as period_from and be in chronological order.

  • version (int | None) – An optional non-negative integer, such as 0 or 3. Do not include the v prefix.

  • file_type (FileType | None) – An optional FileType enum member: FileType.JSON, FileType.CSV, FileType.XML, or FileType.PARQUET. Do not pass a raw string or include the leading period.

  • folders (list[str] | None) – None or a list of optional folder names below the product and data-state folders. Each folder must be non-empty and contain only letters, digits, -, and _. For example, ["on-prem", "revidert_data"].

Return type:

str

Examples

Build a complete path:

>>> create_dataset_path(
...     bucket="bucket",
...     product="ledstill",
...     data_state=DataState.OUTPUT_DATA,
...     short_description="varehandel",
...     period_from="2018-Q1",
...     version=1,
...     file_type=FileType.PARQUET,
... )
'gs://bucket/ledstill/utdata/varehandel_p2018-Q1_v1.parquet'

Build a partial path:

>>> create_dataset_path(product="ledstill", data_state=DataState.INPUT_DATA)
'ledstill/inndata'

Build a filename:

>>> create_dataset_path(short_description="befolkning", period_from="2025", version=0, file_type=FileType.JSON)
'befolkning_p2025_v0.json'

Build a path containing folders:

>>> create_dataset_path(data_state=DataState.OUTPUT_DATA, folders=["publisert", "arkiv"])
'utdata/publisert/arkiv'

Build a filename containing a period range:

>>> create_dataset_path(short_description="handel", period_from="2025-Q1", period_to="2025-Q4", version=2, file_type=FileType.CSV)
'handel_p2025-Q1_p2025-Q4_v2.csv'
Return type:

str

Returns:

A complete GCS path or contiguous relative path fragment.

Raises:

  • TypeError – If an argument has an invalid Python type.

  • ValueError – If an argument does not satisfy the naming standard, the filename is incomplete, the path contains a hierarchy gap, or no path component is supplied.

Parameters:

  • bucket (str | None)

  • product (str | None)

  • data_state (DataState | None)

  • short_description (str | None)

  • period_from (str | None)

  • period_to (str | None)

  • version (int | None)

  • file_type (FileType | None)

  • folders (list[str] | None)

dapla_metadata.standards.name_validator module

class NamingStandardReport(validation_results)[source]

Bases: object

Report object for name standard validation.

Parameters:

validation_results (list[ValidationResult])

evaluate_result()[source]

Returns an appropriate message based on the success rate.

Return type:

str

generate_report()[source]

Format the report as a string.

Return type:

str

success_rate()[source]

Calculate the success rate as a percentage.

Returns:

The success rate as a percentage, or None if no files were validated.

Return type:

int | float | None

class ValidationResult(success, file_path)[source]

Bases: object

Result object for name standard validation.

Parameters:

  • success (bool)

  • file_path (str)

add_message(message)[source]

Add message to list.

Return type:

None

Parameters:

message (str)

add_violation(violation)[source]

Add violation to list.

Return type:

None

Parameters:

violation (str)

to_dict()[source]

Return result as a dictionary.

Return type:

dict

async validate_directory(path)[source]

Validate a file or recursively validate all files in a directory.

Return type:

AsyncGenerator[AsyncGenerator | Task]

Parameters:

path (ReadablePath | VFSPathLike | PathLike[str] | str)

dapla_metadata.standards.standard_validators module

async check_naming_standard(file_path)[source]

Check whether a given path follows the SSB naming standard.

This function checks whether the provided file_path and subdirectories thereof comply with the naming standard. Currently we only examine ‘.parquet’ files. Other files are ignored.

Parameters:

file_path (str | PathLike[str]) –

The path to a bucket, directory, or specific file to validate. This can be in the following forms: - A bucket URL in the form ‘gs://ssb-dapla-felles-data-produkt-test’ - An absolute path to a mounted bucket in the form ‘/buckets/produkt’ - Any subdirectory or file thereof

We also accept paths which don’t yet exist so that you can test if a path will comply.

Returns:

A list of validation results, including success status, checked file path, messages, and any detected violations.

Return type:

list[ValidationResult]

Examples

>>> (await check_naming_standard("/data/example_file.parquet"))[0].success
False

>>> (await check_naming_standard("/buckets/produkt/datadoc/utdata/person-data_p2021_v2.parquet"))[0].success
True
async flatten_generator(gen)[source]

Recursively flatten nested async generators.

Return type:

AsyncGenerator[Task, None]

Parameters:

gen (AsyncGenerator)

generate_validation_report(validation_results)[source]

Generate and print a formatted naming standard validation report.

This function takes a list of ValidationResult objects, creates a NamingStandardReport instance, and prints the generated report.

Parameters:

  • validation_results (list[ValidationResult]) – A list of ValidationResult objects that

  • checks. (contain the outcomes of the name standard)

Returns:

An instance of NamingStandardReport containing the validation results.

Return type:

NamingStandardReport