---
title: Calc With Metadata
marimo-version: 0.24.0
---
# Calculations with metadata
Scope
-----
This guide illustrates how the role of metadata in calculations extends beyond simple filtering.
## Prerequisites
``` {note}
The guide assumes that the SSB Timeseries library is installed and that a working configuration is active.
See [the quickstart guide](quickstart) for instructions to that.
```
The presented functionality relies on `dataset.Dataset` and `meta.taxonomy.Taxonomy`.
Other imports like`types.SeriesType` and external libraries are used only for generating the sample data.
```python {.marimo}
from ssb_timeseries.dataset import Dataset
from ssb_timeseries.meta.taxonomy import Taxonomy
```
Generate sample data
--------------------
```python {.marimo}
from ssb_timeseries.types import SeriesType
from ssb_timeseries.sample_data import create_df
from itertools import product
from datetime import date
```
Generate some test data
```python {.marimo}
def create_some_example_data(
set_name: str,
series_tags: dict[str,list[str]],
):
"""Generate and save some sample data."""
set_tags = { "Country": "Norway" }
df = create_df(
*[value for value in series_tags.values()],
temporality= 'FROM_TO',
start_date="2024-01-01",
end_date="2026-12-01",
freq="MS",
)
Dataset(
name=set_name,
data_type=SeriesType('NONE', 'FROM_TO'),
data=df,
tags = set_tags,
attributes = series_tags.keys(),
).save()
```
We will generate random data for all permutations of some descriptive metadata tags.
This time we include a real classification that we will simply name "taxonomy", and use completely out of context.
(It just happens to have a suitable shape and size.)
```python {.marimo}
taxonomy = Taxonomy(klass_id=157)
taxonomy.print_tree()
```
╙── 0
├─╼ 1
│ ├─╼ 1.1
│ │ ├─╼ 1.1.1
│ │ ├─╼ 1.1.2
│ │ └─╼ 1.1.3
│ └─╼ 1.2
├─╼ 11
│ ├─╼ 11.1
│ └─╼ 11.2
├─╼ 12
│ ├─╼ 12.1
│ │ ├─╼ 12.1.1
│ │ ├─╼ 12.1.10
│ │ ├─╼ 12.1.11
│ │ ├─╼ 12.1.12
│ │ ├─╼ 12.1.13
│ │ ├─╼ 12.1.2
│ │ ├─╼ 12.1.3
│ │ ├─╼ 12.1.4
│ │ ├─╼ 12.1.5
│ │ ├─╼ 12.1.6
│ │ ├─╼ 12.1.7
│ │ ├─╼ 12.1.8
│ │ └─╼ 12.1.9
│ ├─╼ 12.2
│ │ ├─╼ 12.2.1
│ │ ├─╼ 12.2.2
│ │ ├─╼ 12.2.3
│ │ ├─╼ 12.2.4
│ │ └─╼ 12.2.5
│ └─╼ 12.3
│ ├─╼ 12.3.1
│ ├─╼ 12.3.2
│ ├─╼ 12.3.3
│ └─╼ 12.3.4
├─╼ 13
├─╼ 14
├─╼ 15
├─╼ 2
├─╼ 3
├─╼ 4
│ ├─╼ 4.1
│ └─╼ 4.2
├─╼ 5
├─╼ 6
├─╼ 7
│ ├─╼ 7.1
│ ├─╼ 7.2
│ ├─╼ 7.3
│ ├─╼ 7.4
│ ├─╼ 7.5
│ └─╼ 7.6
├─╼ 8
│ ├─╼ 8.1
│ ├─╼ 8.2
│ ├─╼ 8.3
│ ├─╼ 8.4
│ ├─╼ 8.5
│ ├─╼ 8.6
│ ├─╼ 8.7
│ ├─╼ 8.8
│ └─╼ 8.9
└─╼ 9
```python {.marimo}
create_some_example_data(
set_name="More Prices and Volumes",
series_tags = {
"variable": ["price", "volume"],
"product": ["milk", "eggs", "bread", "juice", "ham", "cheese"],
"category": taxonomy.leaf_nodes,
}
)
```
Here we use the 53 `taxonomy.leaf_nodes` to populate a `category` attribute.
Filtering datasets by tags
--------------------------
The most typical use of descriptive metadata, aka `Dataset.tags`, is to extract subsets of datasets for specific purposes.
A simple "example with Prices and Volumes" extracts *prices* and *volumes* for a number of *products* into separate variables and calculate revenues by multiplying them:
```python {.marimo}
prices_and_volumes = Dataset(name="More Prices and Volumes")
prices = prices_and_volumes[{'variable': 'price'}]
volumes = prices_and_volumes[{'variable': 'volume'}]
revenue = prices * volumes
```
The name and tags of the returned dataset need to be updated to make sense:
```python {.marimo}
revenue.rename("More Revenues", ('price', 'revenue'))
revenue.replace_tags(({'variable':'price'}, {'variable': 'revenue'}))
```
So from 53 taxonomy entities times 6 products we get 318 revenue series.
Group by behaviour
------------------
Group by can be configured to run in "auto" mode: using metadata attributes to select whether to calculate sums or averages.
(The functionality was hard coded for the PoC phase. It is now disabled, but a functionality skeleton is still there. The missing link for working properly is configuration interaction.)
```python {.marimo}
volumes.data = volumes.pd # a workaround for BUG
# q = volumes.groupby('Q', 'auto') # --> another bug!
```
## Aggregates
Since we now happen to have a properly tagged dataset containing all the leaf nodes in such a tree, we can calculate the aggregates for the rest of the taxonomy structure, that is for the "parent" nodes of the hierarchy:
```python {.marimo}
taxonomy.parent_nodes
```
['1', '0', '1.1', '11', '12', '12.1', '12.2', '12.3', '4', '7', '8']
```python {.marimo}
list_of_functions = ['sum'] # there are more options --> see the reference
aggregated_revenue = revenue.aggregate(
attributes=["category"], # lengths must match ↓
taxonomies=[taxonomy], # lengths must match ↑
functions=list_of_functions
)
aggregated_revenue.pl.schema.to_python()
```
{'sum(0)': "<class 'float'>",
'sum(1)': "<class 'float'>",
'sum(1.1)': "<class 'float'>",
'sum(11)': "<class 'float'>",
'sum(12)': "<class 'float'>",
'sum(12.1)': "<class 'float'>",
'sum(12.2)': "<class 'float'>",
'sum(12.3)': "<class 'float'>",
'sum(4)': "<class 'float'>",
'sum(7)': "<class 'float'>",
'sum(8)': "<class 'float'>",
'valid_from': "<class 'datetime.datetime'>",
'valid_to': "<class 'datetime.datetime'>"}
```python {.marimo}
# aggregated_revenue.tags
```
Here we can observe a bug: "input" lists data, not series names.
Including input data could be OK elsewhere, but it inflates `.tags` in a way that is not scalable.
(Tags are included in parquet metadata fields, so including data may limit the maximum size of datasets / cause other problems.)
```python {.marimo}
```
Planned: Canonic datasets
-----------------------
Calculations with canonic datasets are sets where a few specific datasets play a key role.
The datasets can reside in local repositories or accessed through API.
Examples:
- Currency conversion.
- Inflation adjustment.
For such features to work, canonic sets and critical attributes must be specified.
Hard coded would work, but configurations would be better.
Planned: Unit conversion
----------------------
Automatic unit conversions based on tags require configurations to identify the name of the unit attribute.