Archiving and sharing

To comply with legal requirements, Statistics Norway commits itself to working according to a formal process model. For the sake of transparency and process reviews, at certain points in the process data has to be persisted. That does not simply mean the data must be saved. Stricter requirements apply. First, immutability: the persisted data must be stored “forever”, without being subject to change. Second, conventions apply to storage formats, naming and documentation.

Data shared between different statistics are subject to the same restrictions.

The conventions that apply are designed for archive and review purposes, not to for efficient data manipulation or retrievel. That is contrary to the purpose of the SSB Timeseries library, which is the reason “archiving” and “sharing” are treated differently from ordinary reads and writes.

Configurations at the set level control how a dataset is archived and shared, but the actual writing happens when data is persisted. Since an archive is a persisted copy, the .archive() function takes care of both.

Setup

data_path = CONFIG.repositories['tutorials']['directory']['options']['path']
def treee():
    print(tree(data_path))
treee()

timeseries/
├── archives/
│   ├── statistics/
│   │   └── PQR/
│   │       └── PQR_p2019-12-31T23-00-00.000+00-00_p2025-08-14T22-00-00.000+00-00_v1.parquet
│   ├── The Sample Statistic/
│   │   └── statistics/
│   │       └── SampleDataset/
│   │           └── SampleDataset_p2021-12-31T23-00-00.000+00-00_p2022-11-30T23-00-00.000+00-00_v1.parquet
│   └── XYZ/
│       └── XYZ_p2025-07-30T22-00-00.000+00-00_p2025-08-05T22-00-00.000+00-00_v2025-08-06T22-00-00.000+00-00_v_v1.parquet
├── AS_OF_AT/
│   └── POPU06/
│       ├── POPU06-as_of_2023-12-31T230000+0000-data.parquet
│       ├── POPU06-as_of_2024-01-31T230000+0000-data.parquet
│       ├── POPU06-as_of_2025-10-31T230000+0000-data.parquet
│       ├── POPU06-as_of_2025-11-30T230000+0000-data.parquet
│       └── ...
├── AS_OF_FROM_TO/
│   ├── BNO/
│   │   ├── BNO-as_of_2025-05-31T220000+0000-data.parquet
│   │   └── BNO-as_of_2025-08-06T220000+0000-data.parquet
│   └── Prices and Volumes/
│       ├── Prices and Volumes-as_of_2023-12-31T230000+0000-data.parquet
│       ├── Prices and Volumes-as_of_2024-01-31T230000+0000-data.parquet
│       ├── Prices and Volumes-as_of_2025-10-31T230000+0000-data.parquet
│       ├── Prices and Volumes-as_of_2025-11-30T230000+0000-data.parquet
│       └── ...
├── metadata/
│   ├── AZ_beverages-metadata.json
│   ├── AZ_drinks-metadata.json
│   ├── SampleDataset-metadata.json
│   ├── XYZ-metadata.json
│   └── ...
├── NONE_AT/
│   ├── PQR/
│   │   └── PQR-latest-data.parquet
│   ├── SampleDataset/
│   │   └── SampleDataset-latest-data.parquet
│   └── XYZ/
│       └── XYZ-latest-data.parquet
├── NONE_FROM_TO/
│   ├── AZ_beverages/
│   │   └── AZ_beverages-latest-data.parquet
│   ├── AZ_drinks/
│   │   └── AZ_drinks-latest-data.parquet
│   └── More Prices and Volumes/
│       └── More Prices and Volumes-latest-data.parquet
└── shared/
    └── default/
        ├── statistics/
        │   └── PQR/
        │       └── PQR_p2019-12-31T23-00-00.000+00-00_p2025-08-14T22-00-00.000+00-00_v1.parquet
        └── XYZ/
            └── XYZ_p2025-07-30T22-00-00.000+00-00_p2025-08-05T22-00-00.000+00-00_v2025-08-06T22-00-00.000+00-00_v_v1.parquet

# what is there before we start?
treee()

timeseries/
├── archives/
│   ├── statistics/
│   │   └── PQR/
│   │       └── PQR_p2019-12-31T23-00-00.000+00-00_p2025-08-14T22-00-00.000+00-00_v1.parquet
│   ├── The Sample Statistic/
│   │   └── statistics/
│   │       └── SampleDataset/
│   │           └── SampleDataset_p2021-12-31T23-00-00.000+00-00_p2022-11-30T23-00-00.000+00-00_v1.parquet
│   └── XYZ/
│       └── XYZ_p2025-07-30T22-00-00.000+00-00_p2025-08-05T22-00-00.000+00-00_v2025-08-06T22-00-00.000+00-00_v_v1.parquet
├── AS_OF_AT/
│   └── POPU06/
│       ├── POPU06-as_of_2023-12-31T230000+0000-data.parquet
│       ├── POPU06-as_of_2024-01-31T230000+0000-data.parquet
│       ├── POPU06-as_of_2025-10-31T230000+0000-data.parquet
│       ├── POPU06-as_of_2025-11-30T230000+0000-data.parquet
│       └── ...
├── AS_OF_FROM_TO/
│   ├── BNO/
│   │   ├── BNO-as_of_2025-05-31T220000+0000-data.parquet
│   │   └── BNO-as_of_2025-08-06T220000+0000-data.parquet
│   └── Prices and Volumes/
│       ├── Prices and Volumes-as_of_2023-12-31T230000+0000-data.parquet
│       ├── Prices and Volumes-as_of_2024-01-31T230000+0000-data.parquet
│       ├── Prices and Volumes-as_of_2025-10-31T230000+0000-data.parquet
│       ├── Prices and Volumes-as_of_2025-11-30T230000+0000-data.parquet
│       └── ...
├── metadata/
│   ├── AZ_beverages-metadata.json
│   ├── AZ_drinks-metadata.json
│   ├── SampleDataset-metadata.json
│   ├── XYZ-metadata.json
│   └── ...
├── NONE_AT/
│   ├── PQR/
│   │   └── PQR-latest-data.parquet
│   ├── SampleDataset/
│   │   └── SampleDataset-latest-data.parquet
│   └── XYZ/
│       └── XYZ-latest-data.parquet
├── NONE_FROM_TO/
│   ├── AZ_beverages/
│   │   └── AZ_beverages-latest-data.parquet
│   ├── AZ_drinks/
│   │   └── AZ_drinks-latest-data.parquet
│   └── More Prices and Volumes/
│       └── More Prices and Volumes-latest-data.parquet
└── shared/
    └── default/
        ├── statistics/
        │   └── PQR/
        │       └── PQR_p2019-12-31T23-00-00.000+00-00_p2025-08-14T22-00-00.000+00-00_v1.parquet
        └── XYZ/
            └── XYZ_p2025-07-30T22-00-00.000+00-00_p2025-08-05T22-00-00.000+00-00_v2025-08-06T22-00-00.000+00-00_v_v1.parquet

Example: point-in-time data, without versioning

from ssb_timeseries.sample_data import xyz_at

import ssb_timeseries as ts

# the ssb archive convention allows no spaces in a name:
set_name = 'SampleDataset'

p = ts.dataset.Dataset(
    name = set_name,
    data_type = ts.types.SeriesType('NONE','AT'),
    data = xyz_at(),
).save()

# what is there after the .save():
treee()

timeseries/
├── archives/
│   ├── statistics/
│   │   └── PQR/
│   │       └── PQR_p2019-12-31T23-00-00.000+00-00_p2025-08-14T22-00-00.000+00-00_v1.parquet
│   ├── The Sample Statistic/
│   │   └── statistics/
│   │       └── SampleDataset/
│   │           └── SampleDataset_p2021-12-31T23-00-00.000+00-00_p2022-11-30T23-00-00.000+00-00_v1.parquet
│   └── XYZ/
│       └── XYZ_p2025-07-30T22-00-00.000+00-00_p2025-08-05T22-00-00.000+00-00_v2025-08-06T22-00-00.000+00-00_v_v1.parquet
├── AS_OF_AT/
│   └── POPU06/
│       ├── POPU06-as_of_2023-12-31T230000+0000-data.parquet
│       ├── POPU06-as_of_2024-01-31T230000+0000-data.parquet
│       ├── POPU06-as_of_2025-10-31T230000+0000-data.parquet
│       ├── POPU06-as_of_2025-11-30T230000+0000-data.parquet
│       └── ...
├── AS_OF_FROM_TO/
│   ├── BNO/
│   │   ├── BNO-as_of_2025-05-31T220000+0000-data.parquet
│   │   └── BNO-as_of_2025-08-06T220000+0000-data.parquet
│   └── Prices and Volumes/
│       ├── Prices and Volumes-as_of_2023-12-31T230000+0000-data.parquet
│       ├── Prices and Volumes-as_of_2024-01-31T230000+0000-data.parquet
│       ├── Prices and Volumes-as_of_2025-10-31T230000+0000-data.parquet
│       ├── Prices and Volumes-as_of_2025-11-30T230000+0000-data.parquet
│       └── ...
├── metadata/
│   ├── AZ_beverages-metadata.json
│   ├── AZ_drinks-metadata.json
│   ├── SampleDataset-metadata.json
│   ├── XYZ-metadata.json
│   └── ...
├── NONE_AT/
│   ├── PQR/
│   │   └── PQR-latest-data.parquet
│   ├── SampleDataset/
│   │   └── SampleDataset-latest-data.parquet
│   └── XYZ/
│       └── XYZ-latest-data.parquet
├── NONE_FROM_TO/
│   ├── AZ_beverages/
│   │   └── AZ_beverages-latest-data.parquet
│   ├── AZ_drinks/
│   │   └── AZ_drinks-latest-data.parquet
│   └── More Prices and Volumes/
│       └── More Prices and Volumes-latest-data.parquet
└── shared/
    └── default/
        ├── statistics/
        │   └── PQR/
        │       └── PQR_p2019-12-31T23-00-00.000+00-00_p2025-08-14T22-00-00.000+00-00_v1.parquet
        └── XYZ/
            └── XYZ_p2025-07-30T22-00-00.000+00-00_p2025-08-05T22-00-00.000+00-00_v2025-08-06T22-00-00.000+00-00_v_v1.parquet

#read the data back, just because we can
q = ts.dataset.Dataset(set_name)

Archiving

Archiving writes to the archive and, for each sharing key, to the destination that key names. The archive path is built from the tags, so product and process_stage are set before the dataset is archived.

statistics_product = 'The Sample Statistic'
q.process_stage = 'statistics'
q.product = statistics_product

q.archive()

print(tree(f'{data_path}/archives'))

archives/
├── statistics/
│   └── PQR/
│       └── PQR_p2019-12-31T23-00-00.000+00-00_p2025-08-14T22-00-00.000+00-00_v1.parquet
├── The Sample Statistic/
│   └── statistics/
│       └── SampleDataset/
│           ├── SampleDataset_p2021-12-31T23-00-00.000+00-00_p2022-11-30T23-00-00.000+00-00_v1.parquet
│           └── SampleDataset_p2021-12-31T23-00-00.000+00-00_p2022-11-30T23-00-00.000+00-00_v2.parquet
└── XYZ/
    └── XYZ_p2025-07-30T22-00-00.000+00-00_p2025-08-05T22-00-00.000+00-00_v2025-08-06T22-00-00.000+00-00_v_v1.parquet

Sharing

Archiving and sharing apply to any dataset, not only the one created above. The two datasets used by the other guides are created here so that this guide runs on its own.

from ssb_timeseries.sample_data import create_df

# Archiving and sharing apply to any dataset, not only the one created above.
# The datasets used by the other guides are created here, from the same
# generators, so that this guide runs on its own and writes identical data.
for name, data in (
    ("XYZ", xyz_at()),
    (
        "PQR",
        create_df(
            ["p", "q", "r"],
            start_date="2020-01-01",
            end_date="2025-06-01",
            freq="D",
            temporality="AT",
        ),
    ),
):
    ts.dataset.Dataset(
        name = name,
        data_type = ts.types.SeriesType('NONE','AT'),
        data = data,
    ).save()

# let us differentiate sharing
r = ts.dataset.Dataset("XYZ")
r.sharing = ["s123", "s234"]
r.archive()

s = ts.dataset.Dataset("PQR")
s.process_stage = "statistics"
s.sharing = ["s234"]
s.archive()

# each sharing key falls back to the destination configured as "default":
print(tree(f'{data_path}/shared'))

shared/
└── default/
    ├── statistics/
    │   └── PQR/
    │       ├── PQR_p2019-12-31T23-00-00.000+00-00_p2025-08-14T22-00-00.000+00-00_v1.parquet
    │       └── PQR_p2019-12-31T23-00-00.000+00-00_p2025-08-14T22-00-00.000+00-00_v2.parquet
    └── XYZ/
        ├── XYZ_p2021-12-31T23-00-00.000+00-00_p2022-11-30T23-00-00.000+00-00_v2.parquet
        └── XYZ_p2025-07-30T22-00-00.000+00-00_p2025-08-05T22-00-00.000+00-00_v2025-08-06T22-00-00.000+00-00_v_v1.parquet