Skip to content

dbt Catalog & project onboarding

This as a fast checkpoint using dbt Catalog to review the project being used in this session. You will then go into the project to onboard properly onto both the base and analytics projects.

  • dbt Catalog prerequisite: production (or staging) deployment environments for both mediapulse_base and mediapulse_analytics, each with at least one successful dbt build job run.
  • Goal: build a real map of where the risk, the gaps, and the open dependencies sit across both projects - not by asking someone, but by reading Catalog's recommendations and the projects' own config and docs.

1. Find the biggest gaps

Open Catalog's Recommendations tab for both projects. Cross-check what it surfaces against your own read of the YAML, and compile a ranked list of the gaps/vulnerabilities you'd flag first. At minimum, look for:

  • a model whose own documentation promises something its access config doesn't actually back up (there's at least one real example of this sitting in mediapulse_base's ads marts)
  • any source or model with no tests at all on what looks like its primary key
  • anywhere mediapulse_analytics depends on mediapulse_base data that isn't actually guaranteed to be reliable yet

You don't need to fix any of these now - later topics in this workshop will have you fix specific ones properly. For now: which three would you raise with the platform team first, and why those three over the others you found?

2. Map the dbt Mesh dependency, and get ahead of streaming events

mediapulse_analytics depends on mediapulse_base through a dbt Mesh project dependency (mediapulse_analytics/dependencies.yml). Using the access: settings in mediapulse_base/dbt_project.yml, confirm which mediapulse_base models mediapulse_analytics is actually allowed to ref() today.

You'll notice the entire staging folder is already access: public. Your assumption is that you will build on their final marts - once the base team has cleaned and transformed the data.

Look at why the fct_streaming_events is not yet ready to be publicly accessed by looking in its description the relative model yaml.

You might also want to note that there are little to no contracts set in the base model - this however will be completed by the other team, you will be able to see the changes in the following session.

Read the base team's actual plan first

mediapulse_base/README.md now has a "Streaming domain availability" section - read it before writing your plan. It tells you what's blocking fct_streaming_events today and the date the base team is committing to.

Write a short plan for what mediapulse_analytics needs to have ready before that date lands, so it isn't a scramble once it does. At minimum, cover:

  • Which specific mediapulse_base streaming model(s)/column(s) would mediapulse_analytics actually want, and why? (You'll firm this up in question 3, below.)
  • mediapulse_analytics already has its own assumptions baked in about "the current streaming platform" - where, and what would need reconciling once a real cross-project ref() exists alongside it? (Hint: the map_streaming_legacy_fields seed.)
  • Beyond "access is public," what would you want confirmed about mediapulse_base's production job before you'd actually build on fct_streaming_events?

3. Legacy vs. new streaming - sketch the consolidation

mediapulse_analytics owns a fully self-contained streamview_legacy domain modeling StreamView, the platform MediaPulse migrated off of. mediapulse_base owns the current platform's streaming domain. Read both sides' source/model YAML and the map_streaming_legacy_fields seed description, then answer:

  • Grain: is a legacy watch record the same grain as a current one? (Look closely at playback_heartbeats vs. usr_watch_events_log - one of these is a heartbeat ping and one is already a completed event with a duration. They are not the same shape.)
  • Fields: which fields exist on one side but not the other (e.g. legacy media_format vs. current ctnt_type; current-only fields like monthly_fee_cents that legacy subscriptions never had)?
  • Identity: using map_streaming_legacy_fields, which legacy subscribers/content can be resolved to a current-platform ID - and what should happen to the ones that can't (is_mapped = false)?

You don't need to build anything yet - just sketch a simple plan: what the target grain of a consolidated fact table would be, which columns come from which side, and where you'd need something like an is_legacy flag to keep the two sources distinguishable downstream. You'll build this for real in Modeling with dbt Mesh.

4. Spot the missing unit test

Open mediapulse_analytics/models/intermediate/int_adv_rep_touchpoint_cumcounts.sql. Can you determine which field is a good candidate for a unit test?

Note:

You don't need to write it yet - list the specific edge-case inputs you'd mock in given, and what you'd expect has_real_text to resolve to for each. You'll write the real thing in Unit testing.

Check what's actually tested on this model today, in _intermediate__models.yml: a dbt_utils.expression_is_true test on the running counts, and a not_null test on has_real_text - nothing that actually pins the classification logic itself down against the tricky inputs it was clearly written to handle (an empty string, pure punctuation, a placeholder like "n/a", a comment just under the length cutoff, mixed case).

Answer:

The has_real_text classification is a case statement built entirely out of regexp_like patterns and string logic on notes_clean - reasoning about it needs no warehouse data at all, just inputs and expected outputs.

Question: Why is a generic data_test the wrong tool?

For validating this specific piece of logic a native dbt unit test gives you something extra here that a data test cannot do alone?

Deliverable: your ranked list of gaps (1), your fct_streaming_events readiness plan (2), your legacy/new consolidation sketch (3), and your list of edge-case unit test inputs (4).