Observing production

What happened after the last change?

A model can work on yesterday's input and fail on today's. Observing production means examining what ran, which checks passed, how long the work took and what data the outputs now represent.

The project uses Elementary to record run and test history. A shared Snowflake Streamlit app presents that history so you can compare failures, model timings and row-count changes. The production reference lists its records and pages. Here we will use the evidence to follow one fictional failure.

Read project health before reading one run

A dbt invocation may select the whole project, one scheduled tag or a small changed subgraph. The latest run is therefore not automatically a statement about the health of every model. A successful intraday run cannot prove that a failing monthly model has recovered; a partial deployment may not include the model whose test failed yesterday.

The app's home page deliberately separates open or recurring issues across runs from issues in the latest build. The first answers "what is currently unhealthy anywhere in the project?" The second explains what happened in the most recent invocation and displays its selection when it was partial. Recent runs then provide the operational sequence connecting those two views.

Follow a failed grain test

Imagine the scheduled build of a person-level activity model. It completes its SQL, but the uniqueness test finds a person twice. The following times and results are illustrative.

Open the failed test and its model history. Compare the failure with recent changes and inspect the returned keys in an approved data environment. Suppose a new input now has two qualifying records per person. That evidence points to the selection or join rule. It does not justify deleting the uniqueness test while the output still promises person grain.

Assess each consumer as well. The skipped dashboard table may be stale, but someone querying the failed model directly can see its new rows. A downstream view can also read those rows through its existing definition. The whole graph has not been rolled back to one consistent earlier state.

Investigate from failure to impact

A useful investigation moves in a consistent order. Beginning with the visible symptom and immediately editing SQL risks treating a skipped descendant or a transient platform error as the root cause.

1. Establish the invocation. Check its command, target, selection and time. Determine whether it was a deploy, scheduled build or deliberately narrow run.
2. Find the first failed node. Separate a model error or failed test from descendants that dbt skipped because their parent did not succeed.
3. Read the history. Decide whether the issue is new, recurring, flaky or part of a longer performance or row-count trend.
4. Inspect the contract and message. Use the model or test detail, then open its SQL, YAML and lineage in the dbt project. Ask whether the data broke an assertion, the implementation broke, or the asserted expectation is no longer correct.
5. Assess impact. Use the DAG to check how many downstream models were skipped and which products consume the affected relation. Existing descendants may still contain their last successful data, which makes staleness part of the incident even when they remain queryable.
6. Recover through the normal workflow.Correct code on a branch and merge it through review. Retry reviewed code deliberately when the cause was transient or upstream data arrived late. Plan a full refresh where an incremental model's history must be repaired. The materialisations guide explains why that history does not change automatically.
7. Confirm resolution. A fix is complete when the relevant model, tests and descendants have succeeded and the expected data freshness has been restored. Merging a PR alone does not establish recovery.

A successful build can still contain old data

Suppose today's source file has not arrived, but a model successfully runs against yesterday's file. Its row count and uniqueness test may look normal. The build succeeded; the output still fails a requirement for today's data.

Compare the source period represented with the consumer's expected period. A run timestamp records execution, not the arrival or completeness of its inputs. A row-count movement is useful evidence but does not explain its cause by itself.

Operational evidence is not semantic proof

The app can show that a model ran successfully, remained near its normal row count and passed every declared test. It cannot prove that an undeclared business assumption is correct. A consistently wrong definition can be perfectly reliable. Observability is strongest when model contracts are clear, tests protect the important decisions and owners respond to changes in the real world as well as failures in the code.

The reverse is also important: not every red result means the data is bad. A source may have legitimately changed, a test may encode the wrong population or a warning may be an accepted temporary condition. The result creates evidence and demands a decision; it does not make that decision automatically.

Ownership closes the loop

Model ownership is not only a documentation convenience. It identifies who can interpret a failure, coordinate affected consumers and decide whether the right response is a data correction, a code change, a revised assertion or an agreed period of stale service. Analysts therefore participate in observability for the models and products whose meaning they own, with engineering support when the cause crosses into ingestion, orchestration, permissions or platform behaviour.

In the example, recovery requires the person model and its tests to pass, the affected consumer to refresh, and the expected source period to be present. Closing the code change alone does not establish those results.