Learn 12~13 min
Observing production
Every dbt run leaves structured evidence about what ran, what failed, how long it took and what was skipped. The project’s Snowflake observability app turns that Elementary history into a shared view of production health.
Deployment is not the end of the work
A green pull request establishes that a proposed change meets the checks run before merge. A successful deployment establishes that the selected production graph built at that moment. Neither guarantees that future source data will continue to satisfy the model's assumptions, that every scheduled lineage will finish on time or that the analytical definition matches every change in the real world.
Observability is how the team learns what the production system is doing after code has been deployed. It connects individual run results into history: a failure can be seen as new or recurring, a slow model can be compared with its normal execution time, a row-count movement can be separated from a build error, and the downstream effect of a failed node can be assessed without reconstructing the run from terminal output.
This is an active part of the analytics development lifecycle. The platform can collect the evidence, but people still decide what healthy means, investigate deviations and feed what they learn back into the next plan and change.
Elementary gives runs a common history
The project records invocation, model and test results in the Elementary schema after dbt runs. Elementary provides dbt-native result and observability tables. Instead of each workflow producing an isolated log that is hard to compare, these records give scheduled builds, deployments and other invocations a common structure. The observability app reads that history from DATA_LAKE__NCL.DBT_OBSERVABILITY.
| Elementary record | What it lets the app answer |
|---|---|
dbt_invocations | Which command, target, warehouse and selection ran, and when? |
dbt_run_results | Which models succeeded, failed or were skipped, and how long did they take? |
elementary_test_results | Which assertions passed, warned or failed, and is the problem recurring? |
dbt_models and dbt_tests | Which project resource, path, SQL and relationship does a result describe? |
ROW_COUNT_LOG | How have materialised table row counts changed over time? |
The app is a Snowflake-native Streamlit application maintained in the snowflake-dbt-observability-streamlit repository. Its queries and thresholds are version-controlled, so the organisation's definition of project health is inspectable and can improve through review.
Read project health before reading one run
A dbt invocation may select the whole project, one scheduled tag or a small changed subgraph. The latest run is therefore not automatically a statement about the health of every model. A successful intraday run cannot prove that a failing monthly model has recovered; a partial deployment may not include the model whose test failed yesterday.
The app's home page deliberately separates open or recurring issues across runs from issues in the latest build. The first answers “what is currently unhealthy anywhere in the project?” The second explains what happened in the most recent invocation and displays its selection when it was partial. Recent runs then provide the operational sequence connecting those two views.
The app supports several kinds of investigation
| Page | Use it to understand |
|---|---|
| Home | Current project health, the latest build and recent invocations |
| Alerts | Active and historical model or test failures, failure streaks and resolution time |
| Models | Models by project path, slow models and individual run histories |
| Tests | Test history, flaky tests and models for which no tests are recorded |
| Runs | Invocation selection, results, duration, skipped nodes and execution timeline |
| Growth | Unexpected growth or shrinkage in materialised table row counts |
| Performance | Total and average execution time, including the models consuming most build time |
Model detail connects operational evidence back to implementation. It exposes recent status, success rate, execution and row-count trends, compiled SQL and applied tests. Test detail shows the assertion's SQL, its run history and its related model. The app is therefore not just a red-light dashboard; it is a route from an observed symptom to the project resource that needs investigation.
Investigate from failure to impact
A useful investigation moves in a consistent order. Beginning with the visible symptom and immediately editing SQL risks treating a skipped descendant or a transient platform error as the root cause.
- Establish the invocation. Check its command, target, selection and time. Determine whether it was a deploy, scheduled build or deliberately narrow run.
- Find the first failed node. Separate a model error or failed test from descendants that dbt skipped because their parent did not succeed.
- Read the history. Decide whether the issue is new, recurring, flaky or part of a longer performance or row-count trend.
- Inspect the contract and message. Use the model or test detail, then open its SQL, YAML and lineage in the dbt project. Ask whether the data broke an assertion, the implementation broke, or the asserted expectation is no longer correct.
- Assess impact. Use the DAG to check how many downstream models were skipped and which products consume the affected relation. Existing descendants may still contain their last successful data, which makes staleness part of the incident even when they remain queryable.
- Recover through the normal workflow.Correct code on a branch and merge it through review. Retry reviewed code deliberately when the cause was transient or upstream data arrived late. Plan a full refresh where an incremental model's history must be repaired. The materialisations guide explains why that history does not change automatically.
- Confirm resolution. A fix is complete when the relevant model, tests and descendants have succeeded and the expected data freshness has been restored—not merely when a PR has merged.
Operational evidence is not semantic proof
The app can show that a model ran successfully, remained near its normal row count and passed every declared test. It cannot prove that an undeclared business assumption is correct. A consistently wrong definition can be perfectly reliable. Observability is strongest when model contracts are clear, tests protect the important decisions and owners respond to changes in the real world as well as failures in the code.
The reverse is also important: not every red result means the data is bad. A source may have legitimately changed, a test may encode the wrong population or a warning may be an accepted temporary condition. The result creates evidence and demands a decision; it does not make that decision automatically.
Ownership closes the loop
Model ownership is not only a documentation convenience. It identifies who can interpret a failure, coordinate affected consumers and decide whether the right response is a data correction, a code change, a revised assertion or an agreed period of stale service. Analysts therefore participate in observability for the models and products whose meaning they own, with engineering support when the cause crosses into ingestion, orchestration, permissions or platform behaviour.
The preceding production lesson explains how code is deployed and scheduled. This page completes that part of the lifecycle: observe what actually happened, understand its effect, recover safely and let the evidence improve the next change.