From merge to production

Release small, useful changes often

We aim to deliver a useful change, check its effect and learn what to do next. A correction to one measure need not wait for an entire dashboard redesign. When the correction is complete, reviewed and validated, it can go through the release process on its own.

This is the practical purpose of agile development here. Work with the people who need the analysis, agree a manageable next outcome and use what you learn to shape the following one. Requirements often become clearer when someone can inspect real columns, definitions and results. Frequent feedback lets us resolve a misunderstanding before it spreads through several models and products.

A small release still needs a complete, defensible definition. An exploratory result with unresolved population rules should stay in development and discussion. Speed comes from reducing how much changes at once and how long feedback takes. The review and validation still need to establish that the proposed result is fit for its intended use.

Split the outcome into independently useful steps

Suppose a service wants a new appointments dashboard. Keeping every source change, measure and visualisation on one branch means the team sees the complete proposal late. A problem with the population then forces revisions across work that already assumes it is correct. A possible sequence is:

Each step includes the SQL, tests and documentation that make it usable. Later work begins from the accepted earlier change. This is an example, not a rule to create one PR per layer. Reuse existing models when they already answer the question. If a producer and its consumer must change together to remain compatible, release that coherent change together.

A breaking change needs a transition. For example, adding a replacement column while retaining the old one can let consumers migrate before a later removal. Agree the meaning and migration with those consumers; renaming a column in a small PR still breaks a query that expects the old name. Small releases reduce the number of changes to reason about, but their effect can still reach many users.

Use each release to test an expectation

Before changing an age boundary, state which records should move and which should remain unchanged. After the focused build, compare those groups against the same inputs and reference date. If the result differs from the expectation, there are fewer changed rules to investigate. Combining that correction with a new join and a different reporting period would make the cause much harder to identify.

Carry that evidence through review, then check the deployment and the affected result. Feedback may confirm the change or reveal a further requirement. Record that next piece of work separately when it can be delivered independently. Frequent releases mean repeating this cycle whenever a useful change is ready, rather than aiming for a fixed number of releases or accumulating unrelated work for a large launch.

The cycle is: agree an outcome, implement it on a branch, validate, review, merge, deploy, inspect the result and decide the next change. A merge is the hand-off to deployment. It is not evidence that the warehouse now contains the intended result. The stages below explain what happens between that decision and the data people use.

Follow one change into production

Suppose a reviewed change corrects a waiting-list count. The SQL and its tests have changed in Git, but production still contains the earlier result. A workflow is an automated job defined in the repository. GitHub Actions runs these jobs on a runner, a machine that checks out the project and executes its commands.

Those are separate results. An approved PR can still fail during deployment, and a successful deployment says nothing about a source delivery that has not arrived yet.

Development builds go to the DEV__ databases

The project's profiles.yml defines development, production and compile-only targets. When you run dbt build with the normal development target, the project naming macros route relations to mirror databases such as DEV__STAGING, DEV__MODELLING and DEV__REPORTING. The data lake has no mirror, so development reads real source data in place while writing only to the DEV__ databases. Production workflows use the prod target and build the supported production relations.

This routing is what keeps normal local and PR builds away from the tables consumers use. Snowflake roles and grants remain the underlying permission boundary. Targets choose destinations; grants decide permissions. In normal work, use the development target locally and let the workflows perform production builds.

Every PR commit passes the compile gate

dbt-compile.yml runs when a relevant pull request is opened, reopened or updated. It executes dbt compile --target ci-dev. This is read-only: it renders the project, reads Snowflake catalogue metadata and analyses the compiled SQL without materialising models.

Fusion includes a SQL engine rather than treating the model body as an opaque string. The compile gate can therefore reject invalid SQL as well as broken ref() and source() calls or invalid YAML and Jinja. Where the target catalogue contains the required metadata, it can also identify unresolved columns and incompatible types. When GitHub creates a merge-queue run, the same workflow compiles with ci-prod so the final queued change is analysed against production metadata before it reaches main.

What compile does not establish is how the query behaves when Snowflake executes it against real data. It does not materialise the relation, count its rows or run its data tests. The merge-queue validation workflow performs that Snowflake build and exercises the model's data contracts.

Snowflake validation runs in the merge queue

Pull-request events report that runtime validation is deferred. Once Merge when ready is selected, GitHub places the change in its merge queue and creates the candidate it would merge into main, including the relevant earlier queued changes. The validation workflow runs against that merge-group commit. It can also be dispatched manually.

A manifest records the project resources and configuration from a run. The workflow compiles against production metadata, fetches the recorded deployment manifest and builds state:modified nodes with the development target. If no manifest exists, it falls back to the dbt nodes changed by the candidate rather than silently running a full build.

When a deployed manifest is available, validation adds --defer --state state --favor-state. Unselected parents then resolve to the production relations recorded in that manifest instead of relying on a complete or fresh DEV mirror. The changed nodes still write only to the DEV__ databases.

A green validation means the selected nodes built and their tests passed with the inputs used by that run. Review still has to establish that the definition, grain and change are the right ones. The Git and pull requests lesson explains how those forms of evidence come together before merge.

A merge deploys the changed graph

A push to main that changes models, macros, seeds, snapshots, packages or project configuration starts dbt-deploy.yml. The workflow fetches the state manifest published by the previous deploy and runs:

dbt build --target prod --select state:modified+ --state state

state:modified selects nodes whose compiled definition has changed. The trailing + also selects every descendant. A change to int_wl_current, for example, deploys that model and the waiting-list facts, provider summaries and published products that depend on it. If no state manifest exists, or the workflow is started manually, it performs a full production build.

The state comparison uses a recorded deployment baseline. Failed runs need explicit investigation; an unrelated later merge does not necessarily retry their failed models. The production reference describes baseline publication and fallback behaviour.

Refreshing data is different from deploying code

After deployment, the waiting-list SQL may stay unchanged for weeks while providers submit newer records. Scheduled builds run the relevant models again so their stored outputs reflect those inputs. A table does not refresh merely because its source changed.

The project has daily, weekly, monthly and intraday selections. Tags and source lineage help determine which models run. Snapshots are included with their consumers in the daily build. The schedule reference contains the exact times and selectors.

Check that the selection covers the model and that its cadence meets the consumer's need. A daily model built successfully at 04:00 can still describe yesterday's source if today's delivery was late. Build time, source period and the user's freshness requirement are different things to inspect.

Failures become GitHub issues

Deploy and scheduled workflows pass their run results to report_dbt_failures.py. A failed scheduled run creates or updates one GitHub issue for that run type and names the failed nodes. Repeated failures add comments to the same issue; a later successful run closes it. The GitHub–Teams subscription carries issues labelled dbt-run-failure into the team channel. SDL intraday is currently excluded from this issue flow.

dbt skips descendants of a failed model or test, so those downstream relations normally retain their previous versions. The failed relation itself still needs inspection. In particular, a model can finish building and then fail a data test, leaving its new relation in place while its dbt descendants are skipped. A red run contains the failure; it does not roll the entire graph back.

The issue links to the run that failed. The next chapter follows the investigation, including how to tell a failed model from its skipped descendants and how to confirm recovery.

What a model author is responsible for

The workflows remove manual deployment and scheduling, but the repository still needs enough information to operate a model correctly. Before merge, make sure the model:

• uses ref() for model dependencies, with source() kept in generated raw models;
• has tests that protect its grain and important business rules;
• has the correct tags for the cadence its consumers require;
• builds in development and has passed the appropriate PR validation;
• documents any unusual operational dependency or recovery step.

After merge, check the deployment result and confirm the expected analytical effect. Give high-impact changes the additional comparisons and consumer communication agreed during review. If the model later appears in a failure issue, its SQL, description, tests, lineage and run artifacts should give the responding analyst enough context to act without reconstructing the original piece of work.

Deployment answers whether the change built. Ongoing health is a separate responsibility: Observing production explains how the Elementary-backed app connects later runs, failures, performance and downstream impact.