SSB Parquedit

A Python package for manually editing tabular data stored as Parquet files on DaplaLab — Statistics Norway’s cloud data platform. Built on top of DuckDB and the DuckLake catalog, it provides a clean, SQL-injection-safe Python interface for creating tables, inserting data, and querying results directly from Google Cloud Storage (GCS). Intended use on single-table editing. Does not support primary- and foreign keys.

---

Features

• Auto-configuration — reads Dapla environment variables to build connection config automatically

• DuckLake catalog integration — metadata stored in PostgreSQL, data stored in GCS

• Create tables from a pandas DataFrame, a JSON Schema dict, or an existing GCS Parquet file

• Insert data from a pandas DataFrame or a gs:// Parquet path — rows are automatically assigned a unique _id (UUID)

• Query tables with structured filters, column selection, sorting, pagination, and multiple output formats (pandas, polars, pyarrow)

• Count rows with optional structured filter conditions

• Check table existence safely

• Partition tables by one or more columns

• SQL injection prevention — all user-supplied filter values are parameterized; column names, table names, and ORDER BY clauses are validated against strict allowlists

---

Requirements

• Python >=3.12

• Access to a DaplaLab environment

• A PostgreSQL instance reachable at localhost for DuckLake metadata storage

• A GCS bucket following the naming convention ssb-{team-name}-data-produkt-{environment}

Python dependencies

Package	Version
duckdb	==1.5.1
pandas	>=3.0.0, <4.0.0
polars	>=1.38.1, <2.0.0
pyarrow	>=23.0.1, <24.0.0
gcsfs	>=2026.1.0, <2027.0.0
click	>=8.0.1

---

Installation

poetry add ssb-parquedit

---

Usage

Basic setup

ParquEdit reads its connection configuration automatically from Dapla-environment variables.

from ssb_parquedit import ParquEdit

# Auto-configure from environment
con = ParquEdit()

Creating a table

Tables can be created from a DataFrame schema, a JSON Schema dict, or an existing Parquet file.

import pandas as pd

df = pd.DataFrame({"name": ["Alice", "Bob"], "age": [30, 25]})

# Create table from DataFrame (empty — schema only)
con.create_table("my_table_1",
                 source=df,
                 product_name="my-product")

# Create and immediately populate with data
con.create_table("my_table_2",
                 source=df,
                 product_name="my-product",
                 fill=True)

# Create from a JSON Schema
schema = {
    "properties": {
        "name": {"type": "string"},
        "age":  {"type": "integer"},
    }
}
con.create_table("my_table_3",
                 source=schema,
                 product_name="my-product")

# Create from an existing GCS Parquet file (schema inferred from file)
con.create_table("my_table_4",
                 source="gs://my-bucket/path/to/file.parquet",
                 product_name="my-product")

# Create with partitioning and immediately populate with data
con.create_table("my_table_5",
                 source=df,
                 product_name="my-product",
                 part_columns=["age"],
                 fill=True)

Note: product_name is required and is stored as a comment on the table. Table names must be lowercase, start with a letter or underscore, and contain only lowercase letters, numbers, and underscores (max 20 characters).

Inserting data in an existing table

# Insert from a DataFrame
con.insert_data("my_table_1",
                 source=df)

# Insert from a GCS Parquet file
con.insert_data("my_table_4",
                 source="gs://my-bucket/path/to/file.parquet")

Each inserted row is automatically assigned a unique _id (UUID string).

Querying data

# View all rows (returns pandas DataFrame by default)
result = con.view("my_table_1")

# Limit and offset (pagination)
result = con.view("my_table_1",
                  limit=10,
                  offset=2)

# Select specific columns
result = con.view("my_table_1",
                  columns=["name", "age"])

# Sort results
result = con.view("my_table_1",
                   order_by="age DESC")

# Return as polars or pyarrow
result = con.view("my_table_1",
                   output_format="polars")

result = con.view("my_table_1",
                   output_format="pyarrow")

Filtering

Filters are structured dicts — never raw SQL strings — ensuring SQL injection safety.

# Single condition
con.view("my_table_1",
         filters={"column": "age", "operator": ">", "value": 25})

# Multiple conditions (implicit AND)
con.view("my_table_1",
        filters=[
            {"column": "age", "operator": ">", "value": 25},
            {"column": "name", "operator": "LIKE", "value": "A%"},
        ])

# Explicit AND / OR
con.view("my_table_1",
        filters={
            "or": [
                {"column": "name", "operator": "=", "value": "Alice"},
                {"column": "name", "operator": "=", "value": "Bob"},
            ]
        })

# IN operator
con.view("my_table_1",
          filters={"column": "age", "operator": "IN", "value": [25, 30, 35]})

# BETWEEN operator
con.view("my_table_1",
          filters={"column": "age", "operator": "BETWEEN", "value": [20, 40]})

# NULL checks
con.view("my_table_1",
          filters={"column": "name", "operator": "IS NOT NULL"})

Supported operators: =, !=, <>, <, >, <=, >=, LIKE, IN, NOT IN, BETWEEN, IS NULL, IS NOT NULL.

Counting rows

total = con.count("my_table_1")
active_adults = con.count("my_table_1",
                            filters=[
                                {"column": "age", "operator": ">=", "value": 18},
                            ])

Checking table existence

if con.exists("my_table_1"):
    print("Table found")

List all tables

con.list_tables()

---

Security

SSB Parquedit is designed with SQL injection prevention as a first-class concern.

Key points:

• All filter values are passed as parameterized query parameters (never interpolated into SQL strings)

• Column names, table names, and ORDER BY clauses are validated against strict allowlists before being used in query construction

• Raw SQL string filters are not accepted

---

Project structure

src/ssb_parquedit/
├── parquedit.py      # ParquEdit facade — main public API
├── connection.py     # DuckDB + DuckLake catalog connection management
├── ddl.py            # DDL operations (CREATE TABLE, partitioning)
├── dml.py            # DML operations (INSERT)
├── query.py          # Query operations (SELECT, COUNT, EXISTS)
├── functions.py      # Environment helpers (Dapla config auto-detection)
└── utils.py          # Schema utilities and SQL sanitization

---

Contributing

Contributions are very welcome. To learn more, see the Contributor Guide.

---

License

Distributed under the terms of the MIT license. SSB Parquedit is free and open source software.

---

Issues

If you encounter any problems, please file an issue along with a detailed description.

---

Credits

This project was generated from Statistics Norway’s SSB PyPI Template. Maintained by Team Fellesfunksjoner at Statistics Norway (Data Enablement Department 724).