Find your source

How source data enters the project

Feeds land in the data lake databases — DATA_LAKE and DATA_LAKE__NCL— without dbt's involvement. The project's first contact with them is the raw layer, and the thing to understand is that nobody writes raw models by hand. There are around seven hundred of them, one per source table, each a near-identical view that quotes the source's columns and renames them to clean snake_case. Hand-writing them would mean seven hundred opportunities for drift and typos; instead, a pipeline generates them.

The pipeline works from a registry — scripts/sources/source_mappings.yml— which lists every database and schema the project reads. When the generation script runs, it queries Snowflake's own metadata for those schemas, writes the source declarations (the YAML that source() calls point at), and writes a raw model for each table. Run it again next month and everything is regenerated to match what Snowflake actually contains.

That regeneration is why the rule about generated files is absolute: an edit to anything under models/raw/ or to an auto_*.yml survives only until the next run. If a raw model is wrong, the fix belongs in the registry, the manual YAML or the generator — the places the pipeline reads from — never in its output.

One wrinkle in the registry matters for the steps below. Most schemas are registered as automatic: every table in them gets a raw model, no decisions needed. A few are marked manual: true, which means tables are opted in one at a time — used for wide schemas where most tables are irrelevant to the project and generating hundreds of unused raw models would be noise. Whether your schema is automatic or manual decides how much work “adding a source” actually is.

First: establish what already exists

Start from the table's full DATABASE.SCHEMA.TABLE address — the schema matters as much as the table name, because it decides the route:

DATA_LAKE__NCL.ANALYST_MANAGED.OPENING_HOURS

Then search the project (Ctrl+P) three ways, in order: the likely staging name (stg_reference_opening_hours — if it exists, your work may already be done; reuse it), the likely raw name (raw_reference_opening_hours — this is the input you will ref()), and finally the bare table name across the repo, which catches a source prefix you did not guess. If the raw model exists, preview it and you are done with this page:

dbt show -s raw_reference_opening_hours

If there is no raw model

Then the table has never been brought into the project, and the registry tells you how big a job that is. Open scripts/sources/source_mappings.yml and search for the schema:

For the manual case, the block you add is small. Find the source in the matching models/sources/manual_*.yml and add your table inside its tables: list, copying a neighbouring block so the indentation is right:

- name: OPENING_HOURS
  identifier: '"OPENING_HOURS"'
  columns:
  - name: SITE_CODE
    data_type: TEXT
  - name: DAY_OF_WEEK
    data_type: NUMBER(2,0)

Run the generation and check what it did

python scripts/sources/run_all_source_generation.py

The script signs into Snowflake (a browser window opens) and runs four stages: it queries the metadata, extracts it, writes the source YAML, and writes the raw models. Because it regenerates from live metadata, its output can include changes you did not cause — a column added to someone else's feed since the last run, for example. So the check afterwards is a genuine review, not a formality:

git status --short
git diff -- models/sources models/raw
dbt parse
dbt show -s raw_reference_opening_hours

You are confirming four things: the expected source YAML changed, your raw model appeared in the right domain folder, the project still parses, and the preview shows the cleaned snake_case columns you will build on. If the diff includes changes to sources you never touched, ask the team about them before bundling them into your PR — they are probably legitimate drift, but that is a decision to make knowingly.

When the input is a seed

Small, team-owned reference data — a mapping of status codes, a list of thresholds — can skip the source machinery entirely and live as a CSV under seeds/. dbt seed loads it into the warehouse and models ref() it like anything else. The boundary: a seed is reference data that belongs to the project's logic and changes by review, the way code does. An extract, a feed, or anything at patient level is not a seed, however convenient the folder looks.

Before writing the model

• Located or generated the raw model
• Reviewed its cleaned columns and sample rows
• Can state the source grain in one sentence
• Confirmed no staging model already exists