Calculations with metadata¶

Scope¶

This guide illustrates how the role of metadata in calculations extends beyond simple filtering.

Prerequisites¶

Note

The guide assumes that the SSB Timeseries library is installed and that a working configuration is active. See the quickstart guide for instructions to that.

The presented functionality relies on dataset.Dataset and meta.taxonomy.Taxonomy. Other imports liketypes.SeriesType and external libraries are used only for generating the sample data.

from ssb_timeseries.dataset import Dataset
from ssb_timeseries.meta.taxonomy import Taxonomy

Generate sample data¶

from ssb_timeseries.types import SeriesType
from ssb_timeseries.sample_data import create_df
from itertools import product
from datetime import date

Generate some test data

def create_some_example_data(
    set_name: str,
    series_tags: dict[str,list[str]],
):
    """Generate and save some sample data."""
    set_tags = { "Country": "Norway" }
    df = create_df(
        *[value for value in series_tags.values()],
        temporality= 'FROM_TO',
        start_date="2024-01-01",
        end_date="2026-12-01",
        freq="MS",
    )
    Dataset(
        name=set_name,
        data_type=SeriesType('NONE', 'FROM_TO'),
        data=df,
        tags = set_tags,
        attributes = series_tags.keys(),
    ).save()

We will generate random data for all permutations of some descriptive metadata tags. This time we include a real classification that we will simply name “taxonomy”, and use completely out of context. (It just happens to have a suitable shape and size.)

taxonomy = Taxonomy(klass_id=157)
taxonomy.print_tree()

╙── 0
    ├─╼ 1
    │   ├─╼ 1.1
    │   │   ├─╼ 1.1.1
    │   │   ├─╼ 1.1.2
    │   │   └─╼ 1.1.3
    │   └─╼ 1.2
    ├─╼ 11
    │   ├─╼ 11.1
    │   └─╼ 11.2
    ├─╼ 12
    │   ├─╼ 12.1
    │   │   ├─╼ 12.1.1
    │   │   ├─╼ 12.1.10
    │   │   ├─╼ 12.1.11
    │   │   ├─╼ 12.1.12
    │   │   ├─╼ 12.1.13
    │   │   ├─╼ 12.1.2
    │   │   ├─╼ 12.1.3
    │   │   ├─╼ 12.1.4
    │   │   ├─╼ 12.1.5
    │   │   ├─╼ 12.1.6
    │   │   ├─╼ 12.1.7
    │   │   ├─╼ 12.1.8
    │   │   └─╼ 12.1.9
    │   ├─╼ 12.2
    │   │   ├─╼ 12.2.1
    │   │   ├─╼ 12.2.2
    │   │   ├─╼ 12.2.3
    │   │   ├─╼ 12.2.4
    │   │   └─╼ 12.2.5
    │   └─╼ 12.3
    │       ├─╼ 12.3.1
    │       ├─╼ 12.3.2
    │       ├─╼ 12.3.3
    │       └─╼ 12.3.4
    ├─╼ 13
    ├─╼ 14
    ├─╼ 15
    ├─╼ 2
    ├─╼ 3
    ├─╼ 4
    │   ├─╼ 4.1
    │   └─╼ 4.2
    ├─╼ 5
    ├─╼ 6
    ├─╼ 7
    │   ├─╼ 7.1
    │   ├─╼ 7.2
    │   ├─╼ 7.3
    │   ├─╼ 7.4
    │   ├─╼ 7.5
    │   └─╼ 7.6
    ├─╼ 8
    │   ├─╼ 8.1
    │   ├─╼ 8.2
    │   ├─╼ 8.3
    │   ├─╼ 8.4
    │   ├─╼ 8.5
    │   ├─╼ 8.6
    │   ├─╼ 8.7
    │   ├─╼ 8.8
    │   └─╼ 8.9
    └─╼ 9

create_some_example_data(
    set_name="More Prices and Volumes",
    series_tags = {
        "variable": ["price", "volume"],
        "product": ["milk", "eggs", "bread", "juice", "ham", "cheese"],
        "category": taxonomy.leaf_nodes,
    }
)

Here we use the 53 taxonomy.leaf_nodes to populate a category attribute.

Filtering datasets by tags¶

The most typical use of descriptive metadata, aka Dataset.tags, is to extract subsets of datasets for specific purposes. A simple “example with Prices and Volumes” extracts prices and volumes for a number of products into separate variables and calculate revenues by multiplying them:

prices_and_volumes = Dataset(name="More Prices and Volumes")
prices = prices_and_volumes[{'variable': 'price'}]
volumes = prices_and_volumes[{'variable': 'volume'}]
revenue = prices * volumes

The name and tags of the returned dataset need to be updated to make sense:

revenue.rename("More Revenues", ('price', 'revenue'))
revenue.replace_tags(({'variable':'price'}, {'variable': 'revenue'}))

So from 53 taxonomy entities times 6 products we get 318 revenue series.

Group by behaviour¶

Group by can be configured to run in “auto” mode: using metadata attributes to select whether to calculate sums or averages.

(The functionality was hard coded for the PoC phase. It is now disabled, but a functionality skeleton is still there. The missing link for working properly is configuration interaction.)

volumes.data = volumes.pd # a workaround for BUG
# q = volumes.groupby('Q', 'auto')  # --> another bug!

Aggregates¶

Since we now happen to have a properly tagged dataset containing all the leaf nodes in such a tree, we can calculate the aggregates for the rest of the taxonomy structure, that is for the “parent” nodes of the hierarchy:

taxonomy.parent_nodes

['1', '0', '1.1', '11', '12', '12.1', '12.2', '12.3', '4', '7', '8']

list_of_functions = ['sum'] # there are more options --> see the reference

aggregated_revenue = revenue.aggregate(
    attributes=["category"],  # lengths must match ↓
    taxonomies=[taxonomy],    # lengths must match ↑
    functions=list_of_functions
)
aggregated_revenue.pl.schema.to_python()

{'sum(0)': "<class 'float'>",
 'sum(1)': "<class 'float'>",
 'sum(1.1)': "<class 'float'>",
 'sum(11)': "<class 'float'>",
 'sum(12)': "<class 'float'>",
 'sum(12.1)': "<class 'float'>",
 'sum(12.2)': "<class 'float'>",
 'sum(12.3)': "<class 'float'>",
 'sum(4)': "<class 'float'>",
 'sum(7)': "<class 'float'>",
 'sum(8)': "<class 'float'>",
 'valid_from': "<class 'datetime.datetime'>",
 'valid_to': "<class 'datetime.datetime'>"}

# aggregated_revenue.tags

Here we can observe a bug: “input” lists data, not series names. Including input data could be OK elsewhere, but it inflates .tags in a way that is not scalable. (Tags are included in parquet metadata fields, so including data may limit the maximum size of datasets / cause other problems.)

Planned: Canonic datasets¶

Calculations with canonic datasets are sets where a few specific datasets play a key role. The datasets can reside in local repositories or accessed through API.

Examples:

Currency conversion.
Inflation adjustment.

For such features to work, canonic sets and critical attributes must be specified. Hard coded would work, but configurations would be better.

Planned: Unit conversion¶

Automatic unit conversions based on tags require configurations to identify the name of the unit attribute.