Archiving and sharing¶
To comply with legal requirements, Statistics Norway commits itself to working according to a formal process model. For the sake of transparency and process reviews, at certain points in the process data has to be persisted. That does not simply mean the data must be saved. Stricter requirements apply. First, immutability: the persisted data must be stored “forever”, without being subject to change. Second, conventions apply to storage formats, naming and documentation.
Data shared between different statistics are subject to the same restrictions.
The conventions that apply are designed for archive and review purposes, not to for efficient data manipulation or retrievel. That is contrary to the purpose of the SSB Timeseries library, which is the reason “archiving” and “sharing” are treated differently from ordinary reads and writes.
Configurations at the set level controls how a dataset is shared, but the actual sharing happens when data is persisted.
Since shared data must be persisted, the .snapshot() function takes care of both.
Setup¶
data_path = CONFIG.repositories['tutorials']['directory']['options']['path']
def treee():
print(tree(data_path))
treee()
timeseries/
├── AS_OF_AT/
│ ├── Sample Data/
│ │ ├── Sample Data-as_of_2023-12-31T230000+0000-data.parquet
│ │ ├── Sample Data-as_of_2024-01-31T230000+0000-data.parquet
│ │ ├── Sample Data-as_of_2024-02-29T230000+0000-data.parquet
│ │ ├── Sample Data-as_of_2024-03-31T220000+0000-data.parquet
│ │ ├── Sample Data-as_of_2024-04-30T220000+0000-data.parquet
│ │ ├── Sample Data-as_of_2024-05-31T220000+0000-data.parquet
│ │ ├── Sample Data-as_of_2024-06-30T220000+0000-data.parquet
│ │ ├── Sample Data-as_of_2024-07-31T220000+0000-data.parquet
│ │ ├── Sample Data-as_of_2024-08-31T220000+0000-data.parquet
│ │ ├── Sample Data-as_of_2024-09-30T220000+0000-data.parquet
│ │ ├── Sample Data-as_of_2024-10-31T230000+0000-data.parquet
│ │ ├── Sample Data-as_of_2024-11-30T230000+0000-data.parquet
│ │ ├── Sample Data-as_of_2024-12-31T230000+0000-data.parquet
│ │ ├── Sample Data-as_of_2025-01-31T230000+0000-data.parquet
│ │ ├── Sample Data-as_of_2025-02-28T230000+0000-data.parquet
│ │ ├── Sample Data-as_of_2025-03-31T220000+0000-data.parquet
│ │ ├── Sample Data-as_of_2025-04-30T220000+0000-data.parquet
│ │ ├── Sample Data-as_of_2025-05-31T220000+0000-data.parquet
│ │ ├── Sample Data-as_of_2025-06-30T220000+0000-data.parquet
│ │ ├── Sample Data-as_of_2025-07-31T220000+0000-data.parquet
│ │ ├── Sample Data-as_of_2025-08-31T220000+0000-data.parquet
│ │ ├── Sample Data-as_of_2025-09-30T220000+0000-data.parquet
│ │ ├── Sample Data-as_of_2025-10-31T230000+0000-data.parquet
│ │ └── Sample Data-as_of_2025-11-30T230000+0000-data.parquet
│ └── XYZ/
│ ├── XYZ-as_of_2025-04-30T220000+0000-data.parquet
│ ├── XYZ-as_of_2025-05-31T220000+0000-data.parquet
│ ├── XYZ-as_of_2025-08-02T220000+0000-data.parquet
│ ├── XYZ-as_of_2025-08-03T220000+0000-data.parquet
│ ├── XYZ-as_of_2025-08-04T220000+0000-data.parquet
│ ├── XYZ-as_of_2025-08-05T220000+0000-data.parquet
│ └── XYZ-as_of_2025-08-06T220000+0000-data.parquet
├── AS_OF_FROM_TO/
│ └── Prices and Volumes/
│ ├── Prices and Volumes-as_of_2023-12-31T230000+0000-data.parquet
│ ├── Prices and Volumes-as_of_2024-01-31T230000+0000-data.parquet
│ ├── Prices and Volumes-as_of_2024-02-29T230000+0000-data.parquet
│ ├── Prices and Volumes-as_of_2024-03-31T220000+0000-data.parquet
│ ├── Prices and Volumes-as_of_2024-04-30T220000+0000-data.parquet
│ ├── Prices and Volumes-as_of_2024-05-31T220000+0000-data.parquet
│ ├── Prices and Volumes-as_of_2024-06-30T220000+0000-data.parquet
│ ├── Prices and Volumes-as_of_2024-07-31T220000+0000-data.parquet
│ ├── Prices and Volumes-as_of_2024-08-31T220000+0000-data.parquet
│ ├── Prices and Volumes-as_of_2024-09-30T220000+0000-data.parquet
│ ├── Prices and Volumes-as_of_2024-10-31T230000+0000-data.parquet
│ ├── Prices and Volumes-as_of_2024-11-30T230000+0000-data.parquet
│ ├── Prices and Volumes-as_of_2024-12-31T230000+0000-data.parquet
│ ├── Prices and Volumes-as_of_2025-01-31T230000+0000-data.parquet
│ ├── Prices and Volumes-as_of_2025-02-28T230000+0000-data.parquet
│ ├── Prices and Volumes-as_of_2025-03-31T220000+0000-data.parquet
│ ├── Prices and Volumes-as_of_2025-04-30T220000+0000-data.parquet
│ ├── Prices and Volumes-as_of_2025-05-31T220000+0000-data.parquet
│ ├── Prices and Volumes-as_of_2025-06-30T220000+0000-data.parquet
│ ├── Prices and Volumes-as_of_2025-07-31T220000+0000-data.parquet
│ ├── Prices and Volumes-as_of_2025-08-31T220000+0000-data.parquet
│ ├── Prices and Volumes-as_of_2025-09-30T220000+0000-data.parquet
│ ├── Prices and Volumes-as_of_2025-10-31T230000+0000-data.parquet
│ └── Prices and Volumes-as_of_2025-11-30T230000+0000-data.parquet
├── metadata/
│ ├── A Sample Dataset-metadata.json
│ ├── AZ Drinks-metadata.json
│ ├── AZ_drikkevarer-metadata.json
│ ├── AZ_drinks-metadata.json
│ ├── AZ_omsetning-metadata.json
│ ├── More Prices and Volumes-metadata.json
│ ├── PQR-metadata.json
│ ├── Prices and Volumes-metadata.json
│ ├── Sample Data-metadata.json
│ └── XYZ-metadata.json
├── NONE_AT/
│ ├── A Sample Dataset/
│ │ └── A Sample Dataset-latest-data.parquet
│ ├── PQR/
│ │ └── PQR-latest-data.parquet
│ └── XYZ/
│ └── XYZ-latest-data.parquet
└── NONE_FROM_TO/
├── AZ Drinks/
│ └── AZ Drinks-latest-data.parquet
├── AZ_drikkevarer/
│ └── AZ_drikkevarer-latest-data.parquet
├── AZ_drinks/
│ └── AZ_drinks-latest-data.parquet
├── AZ_omsetning/
│ └── AZ_omsetning-latest-data.parquet
└── More Prices and Volumes/
└── More Prices and Volumes-latest-data.parquet
# what is there before we start?
treee()
timeseries/
├── AS_OF_AT/
│ ├── Sample Data/
│ │ ├── Sample Data-as_of_2023-12-31T230000+0000-data.parquet
│ │ ├── Sample Data-as_of_2024-01-31T230000+0000-data.parquet
│ │ ├── Sample Data-as_of_2024-02-29T230000+0000-data.parquet
│ │ ├── Sample Data-as_of_2024-03-31T220000+0000-data.parquet
│ │ ├── Sample Data-as_of_2024-04-30T220000+0000-data.parquet
│ │ ├── Sample Data-as_of_2024-05-31T220000+0000-data.parquet
│ │ ├── Sample Data-as_of_2024-06-30T220000+0000-data.parquet
│ │ ├── Sample Data-as_of_2024-07-31T220000+0000-data.parquet
│ │ ├── Sample Data-as_of_2024-08-31T220000+0000-data.parquet
│ │ ├── Sample Data-as_of_2024-09-30T220000+0000-data.parquet
│ │ ├── Sample Data-as_of_2024-10-31T230000+0000-data.parquet
│ │ ├── Sample Data-as_of_2024-11-30T230000+0000-data.parquet
│ │ ├── Sample Data-as_of_2024-12-31T230000+0000-data.parquet
│ │ ├── Sample Data-as_of_2025-01-31T230000+0000-data.parquet
│ │ ├── Sample Data-as_of_2025-02-28T230000+0000-data.parquet
│ │ ├── Sample Data-as_of_2025-03-31T220000+0000-data.parquet
│ │ ├── Sample Data-as_of_2025-04-30T220000+0000-data.parquet
│ │ ├── Sample Data-as_of_2025-05-31T220000+0000-data.parquet
│ │ ├── Sample Data-as_of_2025-06-30T220000+0000-data.parquet
│ │ ├── Sample Data-as_of_2025-07-31T220000+0000-data.parquet
│ │ ├── Sample Data-as_of_2025-08-31T220000+0000-data.parquet
│ │ ├── Sample Data-as_of_2025-09-30T220000+0000-data.parquet
│ │ ├── Sample Data-as_of_2025-10-31T230000+0000-data.parquet
│ │ └── Sample Data-as_of_2025-11-30T230000+0000-data.parquet
│ └── XYZ/
│ ├── XYZ-as_of_2025-04-30T220000+0000-data.parquet
│ ├── XYZ-as_of_2025-05-31T220000+0000-data.parquet
│ ├── XYZ-as_of_2025-08-02T220000+0000-data.parquet
│ ├── XYZ-as_of_2025-08-03T220000+0000-data.parquet
│ ├── XYZ-as_of_2025-08-04T220000+0000-data.parquet
│ ├── XYZ-as_of_2025-08-05T220000+0000-data.parquet
│ └── XYZ-as_of_2025-08-06T220000+0000-data.parquet
├── AS_OF_FROM_TO/
│ └── Prices and Volumes/
│ ├── Prices and Volumes-as_of_2023-12-31T230000+0000-data.parquet
│ ├── Prices and Volumes-as_of_2024-01-31T230000+0000-data.parquet
│ ├── Prices and Volumes-as_of_2024-02-29T230000+0000-data.parquet
│ ├── Prices and Volumes-as_of_2024-03-31T220000+0000-data.parquet
│ ├── Prices and Volumes-as_of_2024-04-30T220000+0000-data.parquet
│ ├── Prices and Volumes-as_of_2024-05-31T220000+0000-data.parquet
│ ├── Prices and Volumes-as_of_2024-06-30T220000+0000-data.parquet
│ ├── Prices and Volumes-as_of_2024-07-31T220000+0000-data.parquet
│ ├── Prices and Volumes-as_of_2024-08-31T220000+0000-data.parquet
│ ├── Prices and Volumes-as_of_2024-09-30T220000+0000-data.parquet
│ ├── Prices and Volumes-as_of_2024-10-31T230000+0000-data.parquet
│ ├── Prices and Volumes-as_of_2024-11-30T230000+0000-data.parquet
│ ├── Prices and Volumes-as_of_2024-12-31T230000+0000-data.parquet
│ ├── Prices and Volumes-as_of_2025-01-31T230000+0000-data.parquet
│ ├── Prices and Volumes-as_of_2025-02-28T230000+0000-data.parquet
│ ├── Prices and Volumes-as_of_2025-03-31T220000+0000-data.parquet
│ ├── Prices and Volumes-as_of_2025-04-30T220000+0000-data.parquet
│ ├── Prices and Volumes-as_of_2025-05-31T220000+0000-data.parquet
│ ├── Prices and Volumes-as_of_2025-06-30T220000+0000-data.parquet
│ ├── Prices and Volumes-as_of_2025-07-31T220000+0000-data.parquet
│ ├── Prices and Volumes-as_of_2025-08-31T220000+0000-data.parquet
│ ├── Prices and Volumes-as_of_2025-09-30T220000+0000-data.parquet
│ ├── Prices and Volumes-as_of_2025-10-31T230000+0000-data.parquet
│ └── Prices and Volumes-as_of_2025-11-30T230000+0000-data.parquet
├── metadata/
│ ├── A Sample Dataset-metadata.json
│ ├── AZ Drinks-metadata.json
│ ├── AZ_drikkevarer-metadata.json
│ ├── AZ_drinks-metadata.json
│ ├── AZ_omsetning-metadata.json
│ ├── More Prices and Volumes-metadata.json
│ ├── PQR-metadata.json
│ ├── Prices and Volumes-metadata.json
│ ├── Sample Data-metadata.json
│ └── XYZ-metadata.json
├── NONE_AT/
│ ├── A Sample Dataset/
│ │ └── A Sample Dataset-latest-data.parquet
│ ├── PQR/
│ │ └── PQR-latest-data.parquet
│ └── XYZ/
│ └── XYZ-latest-data.parquet
└── NONE_FROM_TO/
├── AZ Drinks/
│ └── AZ Drinks-latest-data.parquet
├── AZ_drikkevarer/
│ └── AZ_drikkevarer-latest-data.parquet
├── AZ_drinks/
│ └── AZ_drinks-latest-data.parquet
├── AZ_omsetning/
│ └── AZ_omsetning-latest-data.parquet
└── More Prices and Volumes/
└── More Prices and Volumes-latest-data.parquet
Eksempel: momentane data, uten versjonering¶
from ssb_timeseries.sample_data import xyz_at
from ssb_timeseries.types import SeriesType
from ssb_timeseries.dataset import SeriesType
import ssb_timeseries as ts
set_name = 'A Sample Dataset'
p = ts.dataset.Dataset(
name = set_name,
data_type = ts.types.SeriesType('NONE','AT'),
data = xyz_at(),
).save()
# what is there after the .save():
treee()
timeseries/
├── AS_OF_AT/
│ ├── Sample Data/
│ │ ├── Sample Data-as_of_2023-12-31T230000+0000-data.parquet
│ │ ├── Sample Data-as_of_2024-01-31T230000+0000-data.parquet
│ │ ├── Sample Data-as_of_2024-02-29T230000+0000-data.parquet
│ │ ├── Sample Data-as_of_2024-03-31T220000+0000-data.parquet
│ │ ├── Sample Data-as_of_2024-04-30T220000+0000-data.parquet
│ │ ├── Sample Data-as_of_2024-05-31T220000+0000-data.parquet
│ │ ├── Sample Data-as_of_2024-06-30T220000+0000-data.parquet
│ │ ├── Sample Data-as_of_2024-07-31T220000+0000-data.parquet
│ │ ├── Sample Data-as_of_2024-08-31T220000+0000-data.parquet
│ │ ├── Sample Data-as_of_2024-09-30T220000+0000-data.parquet
│ │ ├── Sample Data-as_of_2024-10-31T230000+0000-data.parquet
│ │ ├── Sample Data-as_of_2024-11-30T230000+0000-data.parquet
│ │ ├── Sample Data-as_of_2024-12-31T230000+0000-data.parquet
│ │ ├── Sample Data-as_of_2025-01-31T230000+0000-data.parquet
│ │ ├── Sample Data-as_of_2025-02-28T230000+0000-data.parquet
│ │ ├── Sample Data-as_of_2025-03-31T220000+0000-data.parquet
│ │ ├── Sample Data-as_of_2025-04-30T220000+0000-data.parquet
│ │ ├── Sample Data-as_of_2025-05-31T220000+0000-data.parquet
│ │ ├── Sample Data-as_of_2025-06-30T220000+0000-data.parquet
│ │ ├── Sample Data-as_of_2025-07-31T220000+0000-data.parquet
│ │ ├── Sample Data-as_of_2025-08-31T220000+0000-data.parquet
│ │ ├── Sample Data-as_of_2025-09-30T220000+0000-data.parquet
│ │ ├── Sample Data-as_of_2025-10-31T230000+0000-data.parquet
│ │ └── Sample Data-as_of_2025-11-30T230000+0000-data.parquet
│ └── XYZ/
│ ├── XYZ-as_of_2025-04-30T220000+0000-data.parquet
│ ├── XYZ-as_of_2025-05-31T220000+0000-data.parquet
│ ├── XYZ-as_of_2025-08-02T220000+0000-data.parquet
│ ├── XYZ-as_of_2025-08-03T220000+0000-data.parquet
│ ├── XYZ-as_of_2025-08-04T220000+0000-data.parquet
│ ├── XYZ-as_of_2025-08-05T220000+0000-data.parquet
│ └── XYZ-as_of_2025-08-06T220000+0000-data.parquet
├── AS_OF_FROM_TO/
│ └── Prices and Volumes/
│ ├── Prices and Volumes-as_of_2023-12-31T230000+0000-data.parquet
│ ├── Prices and Volumes-as_of_2024-01-31T230000+0000-data.parquet
│ ├── Prices and Volumes-as_of_2024-02-29T230000+0000-data.parquet
│ ├── Prices and Volumes-as_of_2024-03-31T220000+0000-data.parquet
│ ├── Prices and Volumes-as_of_2024-04-30T220000+0000-data.parquet
│ ├── Prices and Volumes-as_of_2024-05-31T220000+0000-data.parquet
│ ├── Prices and Volumes-as_of_2024-06-30T220000+0000-data.parquet
│ ├── Prices and Volumes-as_of_2024-07-31T220000+0000-data.parquet
│ ├── Prices and Volumes-as_of_2024-08-31T220000+0000-data.parquet
│ ├── Prices and Volumes-as_of_2024-09-30T220000+0000-data.parquet
│ ├── Prices and Volumes-as_of_2024-10-31T230000+0000-data.parquet
│ ├── Prices and Volumes-as_of_2024-11-30T230000+0000-data.parquet
│ ├── Prices and Volumes-as_of_2024-12-31T230000+0000-data.parquet
│ ├── Prices and Volumes-as_of_2025-01-31T230000+0000-data.parquet
│ ├── Prices and Volumes-as_of_2025-02-28T230000+0000-data.parquet
│ ├── Prices and Volumes-as_of_2025-03-31T220000+0000-data.parquet
│ ├── Prices and Volumes-as_of_2025-04-30T220000+0000-data.parquet
│ ├── Prices and Volumes-as_of_2025-05-31T220000+0000-data.parquet
│ ├── Prices and Volumes-as_of_2025-06-30T220000+0000-data.parquet
│ ├── Prices and Volumes-as_of_2025-07-31T220000+0000-data.parquet
│ ├── Prices and Volumes-as_of_2025-08-31T220000+0000-data.parquet
│ ├── Prices and Volumes-as_of_2025-09-30T220000+0000-data.parquet
│ ├── Prices and Volumes-as_of_2025-10-31T230000+0000-data.parquet
│ └── Prices and Volumes-as_of_2025-11-30T230000+0000-data.parquet
├── metadata/
│ ├── A Sample Dataset-metadata.json
│ ├── AZ Drinks-metadata.json
│ ├── AZ_drikkevarer-metadata.json
│ ├── AZ_drinks-metadata.json
│ ├── AZ_omsetning-metadata.json
│ ├── More Prices and Volumes-metadata.json
│ ├── PQR-metadata.json
│ ├── Prices and Volumes-metadata.json
│ ├── Sample Data-metadata.json
│ └── XYZ-metadata.json
├── NONE_AT/
│ ├── A Sample Dataset/
│ │ └── A Sample Dataset-latest-data.parquet
│ ├── PQR/
│ │ └── PQR-latest-data.parquet
│ └── XYZ/
│ └── XYZ-latest-data.parquet
└── NONE_FROM_TO/
├── AZ Drinks/
│ └── AZ Drinks-latest-data.parquet
├── AZ_drikkevarer/
│ └── AZ_drikkevarer-latest-data.parquet
├── AZ_drinks/
│ └── AZ_drinks-latest-data.parquet
├── AZ_omsetning/
│ └── AZ_omsetning-latest-data.parquet
└── More Prices and Volumes/
└── More Prices and Volumes-latest-data.parquet
#read the data back, just because we can
q = ts.dataset.Dataset(set_name)
…
statistics_product = 'The Sample Statistic'
q.process_stage = 'statistikk'
q.product = statistics_product
#q.snapshot()
CONFIG.configuration_file= '/home/bernhard/code/ssb-timeseries/notebooks/sharing_config.json'
CONFIG.activate()
CONFIG.refresh()
#CONFIG.__dict__
<ssb_timeseries.config.Config object at 0x7fdb9f7c0690>
q.snapshot()
Traceback (most recent call last):
File "/tmp/marimo_516028/__marimo__cell_qnkX_.py", line 1, in
q.snapshot()
~~~~~~~~~~^^
File "/home/bernhard/code/ssb-timeseries/src/ssb_timeseries/dataset.py", line 467, in snapshot
io.persist(self) # is 'archive' a better name than 'persist' or 'snapshot'?
~~~~~~~~~~^^^^^^
File "/home/bernhard/code/ssb-timeseries/src/ssb_timeseries/io/__init__.py", line 437, in persist
date_from = ds.data[ds.datetime_columns].min().min()
~~~~~~~^^^^^^^^^^^^^^^^^^^^^
File "pyarrow/table.pxi", line 1714, in pyarrow.lib._Tabular.__getitem__
File "pyarrow/table.pxi", line 1799, in pyarrow.lib._Tabular.column
return self._column(self._ensure_integer_index(i))
File "pyarrow/table.pxi", line 1745, in pyarrow.lib._Tabular._ensure_integer_index
raise TypeError("Index must either be string or integer")
TypeError: Index must either be string or integer
exception: Index must either be string or integer
print(tree(data_path))
timeseries/
├── AS_OF_AT/
│ ├── Sample Data/
│ │ ├── Sample Data-as_of_2023-12-31T230000+0000-data.parquet
│ │ ├── Sample Data-as_of_2024-01-31T230000+0000-data.parquet
│ │ ├── Sample Data-as_of_2024-02-29T230000+0000-data.parquet
│ │ ├── Sample Data-as_of_2024-03-31T220000+0000-data.parquet
│ │ ├── Sample Data-as_of_2024-04-30T220000+0000-data.parquet
│ │ ├── Sample Data-as_of_2024-05-31T220000+0000-data.parquet
│ │ ├── Sample Data-as_of_2024-06-30T220000+0000-data.parquet
│ │ ├── Sample Data-as_of_2024-07-31T220000+0000-data.parquet
│ │ ├── Sample Data-as_of_2024-08-31T220000+0000-data.parquet
│ │ ├── Sample Data-as_of_2024-09-30T220000+0000-data.parquet
│ │ ├── Sample Data-as_of_2024-10-31T230000+0000-data.parquet
│ │ ├── Sample Data-as_of_2024-11-30T230000+0000-data.parquet
│ │ ├── Sample Data-as_of_2024-12-31T230000+0000-data.parquet
│ │ ├── Sample Data-as_of_2025-01-31T230000+0000-data.parquet
│ │ ├── Sample Data-as_of_2025-02-28T230000+0000-data.parquet
│ │ ├── Sample Data-as_of_2025-03-31T220000+0000-data.parquet
│ │ ├── Sample Data-as_of_2025-04-30T220000+0000-data.parquet
│ │ ├── Sample Data-as_of_2025-05-31T220000+0000-data.parquet
│ │ ├── Sample Data-as_of_2025-06-30T220000+0000-data.parquet
│ │ ├── Sample Data-as_of_2025-07-31T220000+0000-data.parquet
│ │ ├── Sample Data-as_of_2025-08-31T220000+0000-data.parquet
│ │ ├── Sample Data-as_of_2025-09-30T220000+0000-data.parquet
│ │ ├── Sample Data-as_of_2025-10-31T230000+0000-data.parquet
│ │ └── Sample Data-as_of_2025-11-30T230000+0000-data.parquet
│ └── XYZ/
│ ├── XYZ-as_of_2025-04-30T220000+0000-data.parquet
│ ├── XYZ-as_of_2025-05-31T220000+0000-data.parquet
│ ├── XYZ-as_of_2025-08-02T220000+0000-data.parquet
│ ├── XYZ-as_of_2025-08-03T220000+0000-data.parquet
│ ├── XYZ-as_of_2025-08-04T220000+0000-data.parquet
│ ├── XYZ-as_of_2025-08-05T220000+0000-data.parquet
│ └── XYZ-as_of_2025-08-06T220000+0000-data.parquet
├── AS_OF_FROM_TO/
│ └── Prices and Volumes/
│ ├── Prices and Volumes-as_of_2023-12-31T230000+0000-data.parquet
│ ├── Prices and Volumes-as_of_2024-01-31T230000+0000-data.parquet
│ ├── Prices and Volumes-as_of_2024-02-29T230000+0000-data.parquet
│ ├── Prices and Volumes-as_of_2024-03-31T220000+0000-data.parquet
│ ├── Prices and Volumes-as_of_2024-04-30T220000+0000-data.parquet
│ ├── Prices and Volumes-as_of_2024-05-31T220000+0000-data.parquet
│ ├── Prices and Volumes-as_of_2024-06-30T220000+0000-data.parquet
│ ├── Prices and Volumes-as_of_2024-07-31T220000+0000-data.parquet
│ ├── Prices and Volumes-as_of_2024-08-31T220000+0000-data.parquet
│ ├── Prices and Volumes-as_of_2024-09-30T220000+0000-data.parquet
│ ├── Prices and Volumes-as_of_2024-10-31T230000+0000-data.parquet
│ ├── Prices and Volumes-as_of_2024-11-30T230000+0000-data.parquet
│ ├── Prices and Volumes-as_of_2024-12-31T230000+0000-data.parquet
│ ├── Prices and Volumes-as_of_2025-01-31T230000+0000-data.parquet
│ ├── Prices and Volumes-as_of_2025-02-28T230000+0000-data.parquet
│ ├── Prices and Volumes-as_of_2025-03-31T220000+0000-data.parquet
│ ├── Prices and Volumes-as_of_2025-04-30T220000+0000-data.parquet
│ ├── Prices and Volumes-as_of_2025-05-31T220000+0000-data.parquet
│ ├── Prices and Volumes-as_of_2025-06-30T220000+0000-data.parquet
│ ├── Prices and Volumes-as_of_2025-07-31T220000+0000-data.parquet
│ ├── Prices and Volumes-as_of_2025-08-31T220000+0000-data.parquet
│ ├── Prices and Volumes-as_of_2025-09-30T220000+0000-data.parquet
│ ├── Prices and Volumes-as_of_2025-10-31T230000+0000-data.parquet
│ └── Prices and Volumes-as_of_2025-11-30T230000+0000-data.parquet
├── metadata/
│ ├── A Sample Dataset-metadata.json
│ ├── AZ Drinks-metadata.json
│ ├── AZ_drikkevarer-metadata.json
│ ├── AZ_drinks-metadata.json
│ ├── AZ_omsetning-metadata.json
│ ├── More Prices and Volumes-metadata.json
│ ├── PQR-metadata.json
│ ├── Prices and Volumes-metadata.json
│ ├── Sample Data-metadata.json
│ └── XYZ-metadata.json
├── NONE_AT/
│ ├── A Sample Dataset/
│ │ └── A Sample Dataset-latest-data.parquet
│ ├── PQR/
│ │ └── PQR-latest-data.parquet
│ └── XYZ/
│ └── XYZ-latest-data.parquet
└── NONE_FROM_TO/
├── AZ Drinks/
│ └── AZ Drinks-latest-data.parquet
├── AZ_drikkevarer/
│ └── AZ_drikkevarer-latest-data.parquet
├── AZ_drinks/
│ └── AZ_drinks-latest-data.parquet
├── AZ_omsetning/
│ └── AZ_omsetning-latest-data.parquet
└── More Prices and Volumes/
└── More Prices and Volumes-latest-data.parquet
# let us differentiate sharing
r = ts.dataset.Dataset("XYZ")
r.sharing = [{'team': 's123', 'path': f'/home/bernhard/timeseries/{statistics_product}/shared/s123/'}, {'team': 's234', 'path': f'/home/bernhard/timeseries/{statistics_product}/shared/s234/'}]
r.snapshot()
treee()
Traceback (most recent call last):
File "/tmp/marimo_516028/__marimo__cell_Vxnm_.py", line 4, in
r.snapshot()
~~~~~~~~~~^^
File "/home/bernhard/code/ssb-timeseries/src/ssb_timeseries/dataset.py", line 467, in snapshot
io.persist(self) # is 'archive' a better name than 'persist' or 'snapshot'?
~~~~~~~~~~^^^^^^
File "/home/bernhard/code/ssb-timeseries/src/ssb_timeseries/io/__init__.py", line 437, in persist
date_from = ds.data[ds.datetime_columns].min().min()
~~~~~~~^^^^^^^^^^^^^^^^^^^^^
File "pyarrow/table.pxi", line 1714, in pyarrow.lib._Tabular.__getitem__
return self.column(key)
File "pyarrow/table.pxi", line 1799, in pyarrow.lib._Tabular.column
return self._column(self._ensure_integer_index(i))
File "pyarrow/table.pxi", line 1745, in pyarrow.lib._Tabular._ensure_integer_index
raise TypeError("Index must either be string or integer")
TypeError: Index must either be string or integer
exception: Index must either be string or integer
s = ts.dataset.Dataset("PQR")
s.process_stage = 'statistikk'
s.sharing = [
{"team": "s234", "path": f'/home/bernhard/timeseries/{statistics_product}/shared/s234/'},
]
s.snapshot()
Traceback (most recent call last):
File "/tmp/marimo_516028/__marimo__cell_DnEU_.py", line 6, in
s.snapshot()
~~~~~~~~~~^^
File "/home/bernhard/code/ssb-timeseries/src/ssb_timeseries/dataset.py", line 467, in snapshot
io.persist(self) # is 'archive' a better name than 'persist' or 'snapshot'?
~~~~~~~~~~^^^^^^
File "/home/bernhard/code/ssb-timeseries/src/ssb_timeseries/io/__init__.py", line 437, in persist
date_from = ds.data[ds.datetime_columns].min().min()
~~~~~~~^^^^^^^^^^^^^^^^^^^^^
File "pyarrow/table.pxi", line 1714, in pyarrow.lib._Tabular.__getitem__
return self.column(key)
File "pyarrow/table.pxi", line 1799, in pyarrow.lib._Tabular.column
return self._column(self._ensure_integer_index(i))
File "pyarrow/table.pxi", line 1745, in pyarrow.lib._Tabular._ensure_integer_index
raise TypeError("Index must either be string or integer")
TypeError: Index must either be string or integer
exception: Index must either be string or integer
treee()
timeseries/
├── AS_OF_AT/
│ ├── Sample Data/
│ │ ├── Sample Data-as_of_2023-12-31T230000+0000-data.parquet
│ │ ├── Sample Data-as_of_2024-01-31T230000+0000-data.parquet
│ │ ├── Sample Data-as_of_2024-02-29T230000+0000-data.parquet
│ │ ├── Sample Data-as_of_2024-03-31T220000+0000-data.parquet
│ │ ├── Sample Data-as_of_2024-04-30T220000+0000-data.parquet
│ │ ├── Sample Data-as_of_2024-05-31T220000+0000-data.parquet
│ │ ├── Sample Data-as_of_2024-06-30T220000+0000-data.parquet
│ │ ├── Sample Data-as_of_2024-07-31T220000+0000-data.parquet
│ │ ├── Sample Data-as_of_2024-08-31T220000+0000-data.parquet
│ │ ├── Sample Data-as_of_2024-09-30T220000+0000-data.parquet
│ │ ├── Sample Data-as_of_2024-10-31T230000+0000-data.parquet
│ │ ├── Sample Data-as_of_2024-11-30T230000+0000-data.parquet
│ │ ├── Sample Data-as_of_2024-12-31T230000+0000-data.parquet
│ │ ├── Sample Data-as_of_2025-01-31T230000+0000-data.parquet
│ │ ├── Sample Data-as_of_2025-02-28T230000+0000-data.parquet
│ │ ├── Sample Data-as_of_2025-03-31T220000+0000-data.parquet
│ │ ├── Sample Data-as_of_2025-04-30T220000+0000-data.parquet
│ │ ├── Sample Data-as_of_2025-05-31T220000+0000-data.parquet
│ │ ├── Sample Data-as_of_2025-06-30T220000+0000-data.parquet
│ │ ├── Sample Data-as_of_2025-07-31T220000+0000-data.parquet
│ │ ├── Sample Data-as_of_2025-08-31T220000+0000-data.parquet
│ │ ├── Sample Data-as_of_2025-09-30T220000+0000-data.parquet
│ │ ├── Sample Data-as_of_2025-10-31T230000+0000-data.parquet
│ │ └── Sample Data-as_of_2025-11-30T230000+0000-data.parquet
│ └── XYZ/
│ ├── XYZ-as_of_2025-04-30T220000+0000-data.parquet
│ ├── XYZ-as_of_2025-05-31T220000+0000-data.parquet
│ ├── XYZ-as_of_2025-08-02T220000+0000-data.parquet
│ ├── XYZ-as_of_2025-08-03T220000+0000-data.parquet
│ ├── XYZ-as_of_2025-08-04T220000+0000-data.parquet
│ ├── XYZ-as_of_2025-08-05T220000+0000-data.parquet
│ └── XYZ-as_of_2025-08-06T220000+0000-data.parquet
├── AS_OF_FROM_TO/
│ └── Prices and Volumes/
│ ├── Prices and Volumes-as_of_2023-12-31T230000+0000-data.parquet
│ ├── Prices and Volumes-as_of_2024-01-31T230000+0000-data.parquet
│ ├── Prices and Volumes-as_of_2024-02-29T230000+0000-data.parquet
│ ├── Prices and Volumes-as_of_2024-03-31T220000+0000-data.parquet
│ ├── Prices and Volumes-as_of_2024-04-30T220000+0000-data.parquet
│ ├── Prices and Volumes-as_of_2024-05-31T220000+0000-data.parquet
│ ├── Prices and Volumes-as_of_2024-06-30T220000+0000-data.parquet
│ ├── Prices and Volumes-as_of_2024-07-31T220000+0000-data.parquet
│ ├── Prices and Volumes-as_of_2024-08-31T220000+0000-data.parquet
│ ├── Prices and Volumes-as_of_2024-09-30T220000+0000-data.parquet
│ ├── Prices and Volumes-as_of_2024-10-31T230000+0000-data.parquet
│ ├── Prices and Volumes-as_of_2024-11-30T230000+0000-data.parquet
│ ├── Prices and Volumes-as_of_2024-12-31T230000+0000-data.parquet
│ ├── Prices and Volumes-as_of_2025-01-31T230000+0000-data.parquet
│ ├── Prices and Volumes-as_of_2025-02-28T230000+0000-data.parquet
│ ├── Prices and Volumes-as_of_2025-03-31T220000+0000-data.parquet
│ ├── Prices and Volumes-as_of_2025-04-30T220000+0000-data.parquet
│ ├── Prices and Volumes-as_of_2025-05-31T220000+0000-data.parquet
│ ├── Prices and Volumes-as_of_2025-06-30T220000+0000-data.parquet
│ ├── Prices and Volumes-as_of_2025-07-31T220000+0000-data.parquet
│ ├── Prices and Volumes-as_of_2025-08-31T220000+0000-data.parquet
│ ├── Prices and Volumes-as_of_2025-09-30T220000+0000-data.parquet
│ ├── Prices and Volumes-as_of_2025-10-31T230000+0000-data.parquet
│ └── Prices and Volumes-as_of_2025-11-30T230000+0000-data.parquet
├── metadata/
│ ├── A Sample Dataset-metadata.json
│ ├── AZ Drinks-metadata.json
│ ├── AZ_drikkevarer-metadata.json
│ ├── AZ_drinks-metadata.json
│ ├── AZ_omsetning-metadata.json
│ ├── More Prices and Volumes-metadata.json
│ ├── PQR-metadata.json
│ ├── Prices and Volumes-metadata.json
│ ├── Sample Data-metadata.json
│ └── XYZ-metadata.json
├── NONE_AT/
│ ├── A Sample Dataset/
│ │ └── A Sample Dataset-latest-data.parquet
│ ├── PQR/
│ │ └── PQR-latest-data.parquet
│ └── XYZ/
│ └── XYZ-latest-data.parquet
└── NONE_FROM_TO/
├── AZ Drinks/
│ └── AZ Drinks-latest-data.parquet
├── AZ_drikkevarer/
│ └── AZ_drikkevarer-latest-data.parquet
├── AZ_drinks/
│ └── AZ_drinks-latest-data.parquet
├── AZ_omsetning/
│ └── AZ_omsetning-latest-data.parquet
└── More Prices and Volumes/
└── More Prices and Volumes-latest-data.parquet
timeseries/
├── AS_OF_AT/
│ ├── Sample Data/
│ │ ├── Sample Data-as_of_2023-12-31T230000+0000-data.parquet
│ │ ├── Sample Data-as_of_2024-01-31T230000+0000-data.parquet
│ │ ├── Sample Data-as_of_2024-02-29T230000+0000-data.parquet
│ │ ├── Sample Data-as_of_2024-03-31T220000+0000-data.parquet
│ │ ├── Sample Data-as_of_2024-04-30T220000+0000-data.parquet
│ │ ├── Sample Data-as_of_2024-05-31T220000+0000-data.parquet
│ │ ├── Sample Data-as_of_2024-06-30T220000+0000-data.parquet
│ │ ├── Sample Data-as_of_2024-07-31T220000+0000-data.parquet
│ │ ├── Sample Data-as_of_2024-08-31T220000+0000-data.parquet
│ │ ├── Sample Data-as_of_2024-09-30T220000+0000-data.parquet
│ │ ├── Sample Data-as_of_2024-10-31T230000+0000-data.parquet
│ │ ├── Sample Data-as_of_2024-11-30T230000+0000-data.parquet
│ │ ├── Sample Data-as_of_2024-12-31T230000+0000-data.parquet
│ │ ├── Sample Data-as_of_2025-01-31T230000+0000-data.parquet
│ │ ├── Sample Data-as_of_2025-02-28T230000+0000-data.parquet
│ │ ├── Sample Data-as_of_2025-03-31T220000+0000-data.parquet
│ │ ├── Sample Data-as_of_2025-04-30T220000+0000-data.parquet
│ │ ├── Sample Data-as_of_2025-05-31T220000+0000-data.parquet
│ │ ├── Sample Data-as_of_2025-06-30T220000+0000-data.parquet
│ │ ├── Sample Data-as_of_2025-07-31T220000+0000-data.parquet
│ │ ├── Sample Data-as_of_2025-08-31T220000+0000-data.parquet
│ │ ├── Sample Data-as_of_2025-09-30T220000+0000-data.parquet
│ │ ├── Sample Data-as_of_2025-10-31T230000+0000-data.parquet
│ │ └── Sample Data-as_of_2025-11-30T230000+0000-data.parquet
│ └── XYZ/
│ ├── XYZ-as_of_2025-04-30T220000+0000-data.parquet
│ ├── XYZ-as_of_2025-05-31T220000+0000-data.parquet
│ ├── XYZ-as_of_2025-08-02T220000+0000-data.parquet
│ ├── XYZ-as_of_2025-08-03T220000+0000-data.parquet
│ ├── XYZ-as_of_2025-08-04T220000+0000-data.parquet
│ ├── XYZ-as_of_2025-08-05T220000+0000-data.parquet
│ └── XYZ-as_of_2025-08-06T220000+0000-data.parquet
├── AS_OF_FROM_TO/
│ └── Prices and Volumes/
│ ├── Prices and Volumes-as_of_2023-12-31T230000+0000-data.parquet
│ ├── Prices and Volumes-as_of_2024-01-31T230000+0000-data.parquet
│ ├── Prices and Volumes-as_of_2024-02-29T230000+0000-data.parquet
│ ├── Prices and Volumes-as_of_2024-03-31T220000+0000-data.parquet
│ ├── Prices and Volumes-as_of_2024-04-30T220000+0000-data.parquet
│ ├── Prices and Volumes-as_of_2024-05-31T220000+0000-data.parquet
│ ├── Prices and Volumes-as_of_2024-06-30T220000+0000-data.parquet
│ ├── Prices and Volumes-as_of_2024-07-31T220000+0000-data.parquet
│ ├── Prices and Volumes-as_of_2024-08-31T220000+0000-data.parquet
│ ├── Prices and Volumes-as_of_2024-09-30T220000+0000-data.parquet
│ ├── Prices and Volumes-as_of_2024-10-31T230000+0000-data.parquet
│ ├── Prices and Volumes-as_of_2024-11-30T230000+0000-data.parquet
│ ├── Prices and Volumes-as_of_2024-12-31T230000+0000-data.parquet
│ ├── Prices and Volumes-as_of_2025-01-31T230000+0000-data.parquet
│ ├── Prices and Volumes-as_of_2025-02-28T230000+0000-data.parquet
│ ├── Prices and Volumes-as_of_2025-03-31T220000+0000-data.parquet
│ ├── Prices and Volumes-as_of_2025-04-30T220000+0000-data.parquet
│ ├── Prices and Volumes-as_of_2025-05-31T220000+0000-data.parquet
│ ├── Prices and Volumes-as_of_2025-06-30T220000+0000-data.parquet
│ ├── Prices and Volumes-as_of_2025-07-31T220000+0000-data.parquet
│ ├── Prices and Volumes-as_of_2025-08-31T220000+0000-data.parquet
│ ├── Prices and Volumes-as_of_2025-09-30T220000+0000-data.parquet
│ ├── Prices and Volumes-as_of_2025-10-31T230000+0000-data.parquet
│ └── Prices and Volumes-as_of_2025-11-30T230000+0000-data.parquet
├── metadata/
│ ├── A Sample Dataset-metadata.json
│ ├── AZ Drinks-metadata.json
│ ├── AZ_drikkevarer-metadata.json
│ ├── AZ_drinks-metadata.json
│ ├── AZ_omsetning-metadata.json
│ ├── More Prices and Volumes-metadata.json
│ ├── PQR-metadata.json
│ ├── Prices and Volumes-metadata.json
│ ├── Sample Data-metadata.json
│ └── XYZ-metadata.json
├── NONE_AT/
│ ├── A Sample Dataset/
│ │ └── A Sample Dataset-latest-data.parquet
│ ├── PQR/
│ │ └── PQR-latest-data.parquet
│ └── XYZ/
│ └── XYZ-latest-data.parquet
└── NONE_FROM_TO/
├── AZ Drinks/
│ └── AZ Drinks-latest-data.parquet
├── AZ_drikkevarer/
│ └── AZ_drikkevarer-latest-data.parquet
├── AZ_drinks/
│ └── AZ_drinks-latest-data.parquet
├── AZ_omsetning/
│ └── AZ_omsetning-latest-data.parquet
└── More Prices and Volumes/
└── More Prices and Volumes-latest-data.parquet