Skip to content

THE DATA FILE FORMAT

This document is a reference for the CSV data file that accompanies each dataset's metadata file. It describes the file format and the validation rules applied to it. For a step-by-step walkthrough of preparing, validating and packaging a full dataset, see Getting started. For a full reference of the metadata JSON file, see the metadata model.

File format

A data file must be supplied as a csv file with semicolon as the column seperator, with no header row. There must always be 5 columns present in this order:

  1. identifier
  2. measure
  3. start date
  4. stop date
  5. empty column (This column is reserved for an extra attribute variable if that is considered necessary. Example: Datasource)

For a dataset with temporalityType FIXED, the value does not change over time, so there is no time period to describe — both the start and stop columns should be left empty. On microdata.no, this is reflected by showing the variable's validity period as infinite ("∞"). See Validation rules by temporality type below for details on stop-column backwards compatibility.

Example: 

12345678910;1;;;
12345678911;2;;;
12345678912;1;;;

This dataset describes the sex of a group of persons. The columns can be described like this: * Identifier: FNR * Measure: Sex * Start: empty * Stop: empty

Example: 

12345678910;100000;2020-01-01;2020-12-31;
12345678910;200000;2021-01-01;2021-12-31;
12345678911;100000;2018-01-01;2018-12-31;
12345678911;150000;2020-01-01;2020-12-31;

This dataset describes a group of persons gross income accumulated yearly. The columns can be described like this: * Identifier: FNR * Measure: Accumulated gross income for the time period * Start: start of time period * Stop: end of time period * Empty column (This column is reserved for an extra attribute variable if that is considered necessary. As there is no need here, it remains empty.)

General validation rules

  • There can be no empty rows in the dataset
  • Every row must have exactly 5 fields (4 semicolons), even if the last one or more are left empty
  • Every row must have a non-empty identifier
  • Every row must have a non-empty measure
  • Values in the stop- and start-columns must be formatted correctly: "YYYY-MM-DD". Example "2020-12-31".
  • DOUBLE values must use . (period) as the decimal separator — not , (comma)
  • The data file must be utf-8 encoded

Validation rules by temporality type

  • FIXED (Constant value, ex.: place of birth)
    • All rows must have a unique identifier (no repeating identifiers within a dataset)
    • The start-column must be empty. A non-empty start date is a validation error.
    • The stop-column is ignored and should be left empty. Non-empty stop dates are accepted for backwards compatibility and produce a deprecation warning, but the values are discarded and never used. Support for non-empty stop dates will be removed in a future version.
  • STATUS (measurement taken at a certain point in time. (cross section))
    • All rows must have a start date
    • All rows must have a stop date
    • Start and stop date must be equal for any given row
  • ACCUMULATED (Accumulated over a period. Ex.: yearly income)
    • All rows must have a start date
    • All rows must have a stop date
    • Start can not be later than stop
    • Time periods for the same identifiers must not intersect
  • EVENT (data state for validity period)
    • All rows must have a start date
    • If there is a non-empty value in the stop column for a given row; start can not be later than stop
    • Time periods for the same identifiers must not intersect (A row without a stop date is considered an ongoing event, and will intersect with all timespans after its start date)