From open data to a governed catalog: Statbel, Databricks, BigQuery and OpenMetadata

In short

I took the open data of the Belgian statistical office (Statbel) and built, end to end, the physical layer that a data catalog needs: the files are loaded, cleaned and typed; every table and every column is described from the official dictionaries; keys are tested before they are declared; quality is measured and reported; and the result is published to BigQuery and harvested by OpenMetadata, next to a business glossary. Nothing is hidden: what the data does not say, the pipeline says it does not say.

This is a demonstration and training project, not a production system. It runs on Databricks Free Edition and stays within the BigQuery free tier.

Why I built it

It is the physical layer of my data catalog training: Data Owners and Data Stewards learn to link real columns to business terms. For that exercise they need a catalog that looks like the real thing, with real names, real ambiguity and real quality problems, and without confidential data. It is also the kind of foundation I put in place for clients: the same questions of ownership, definitions, keys, quality and access, answered in code that anyone can read.

The architecture

Governance, built into the pipeline

What the data taught me

The checks found facts about the real files that a Data Steward has to know, document and decide on. None of them is a mistake of Statbel. Here are a few of the fifteen the pipeline records.

Unity Catalog or OpenMetadata?

Both, for different jobs. The glossary says what the data means and who approved it; the platform enforces who may use it. I connected both on purpose, to compare them on the same tables.

What went wrong, and what I learned

The most useful part of a project like this is what does not work the first time. A selection, from the connection of the catalogs.

Explore the result

The code

Everything is public: github.com/OlivierBouchezDataGovernance/statbel-lakehouse. Notebooks 00 to 07 run in order on a free Databricks workspace; the README and the install guide explain the Google Cloud and OpenMetadata side. Two short extracts show the style.

# config/datasets.yml: nothing else is needed to load, document and publish this table
- table: pop_structure
  domain: pop
  title: Population by municipality, nationality, marital status, age and sex
  period: "2026-01-01"
  sources:
    - url: https://statbel.fgov.be/.../TF_SOC_POP_STRUCT_2026.zip
  dictionary: https://statbel.fgov.be/.../Columns%20description.xlsx
  dictionary_aliases:           # the dictionary names the column differently
    CD_REFNIS: CD_MUNTY_REFNIS
  primary_key: [CD_REFNIS, CD_SEX, CD_NATLTY, CD_CIV_STS, CD_AGE]
  foreign_keys:
    - {column: CD_REFNIS, references: ref_municipality, referenced_column: CD_REFNIS}

# Statbel naming convention, used as the default typing rule
CD_  code     -> string   # codes keep their leading zeros
TX_  label    -> string
MS_  measure  -> bigint, or double if decimals are found
DT_  date     -> date     # dd/MM/yyyy or yyyy-MM-dd
FL_  flag     -> boolean
LVL_ level    -> int
XY_  coord.   -> double

Limits and licence

• It is a demonstration and training project: it runs in your own accounts, at your own risk and cost.
• Databricks Free Edition has no system tables, so there is no automatic lineage or usage; the lineage across the two engines (Databricks to BigQuery) is not detected automatically either.
• Statbel data: statbel.fgov.be/en/open-data, reused under CC BY 4.0. The definitions and structure are my own analysis; this work is not affiliated with Statbel.
• Code and content: CC BY-NC 4.0, free for non-commercial use with attribution.

Let’s talk about your data

Documented, tested and measurable data is what makes a catalog useful, and what AI needs to be trustworthy. If you want the same foundations in your context, or a training for your Data Owners and Stewards, get in touch.

Contact me The data catalog training My four-phase approach