Skip to content

dbt Catalog & Review

You own the streaming domain in mediapulse_base. That makes you a producer, not just someone writing models for yourself: other teams build on what you ship. Your biggest consumer today is mediapulse_analytics, and the point of the next few topics is serving them with something they can actually trust, not just something that happens to compile. This is a fast checkpoint using dbt Catalog, then straight into onboarding onto what you actually own and what's already depending on it.

Prerequisite: production (or staging) deployment environments for both mediapulse_base and mediapulse_analytics, each with at least one successful dbt build job run.

Goal: find out exactly what you're already responsible for: what's public, what's actually being used, and where the biggest gap between what your project claims and what it delivers currently sits, before anyone asks you to build on it for real.

1. Find the biggest gaps

Open Catalog's Recommendations tab for mediapulse_base. Compare what it flags to what you find by reading the YAML yourself, then write a ranked list of the gaps and risks you'd raise first. At minimum, look for:

  • a model whose own documentation promises something its access config doesn't actually back up (there's at least one real example of this sitting in mediapulse_base's ads marts)
  • any source or model with no tests at all on what looks like its primary key
  • anywhere your own docs claim something is safe to build on when it isn't yet (you'll find the biggest one of these in step 3)

You don't need to fix any of these now. Later topics will have you fix specific ones properly. For now: which three would you raise with your own team first, and why those three over the others you found?

2. Map who can reach what, and who actually does

Being a producer means knowing exactly what you expose to other teams. Look at the lineage of your project - can you see which models are being consumed by the other project?

Access rights: Now that you know which models are used, can you see where you set the model visibility? dbt Mesh gates cross-project ref()s with a config: access: setting on every model - public (any project can ref() it), protected (only within this project), or private (only within a matching group - not something you need yet, that's a group3 topic). You don't need the rest of Mesh's mechanics for this exercise, just enough of the syntax to answer the questions below.

Using the access: settings in mediapulse_base/dbt_project.yml, list every model in your project that's currently access: public. Staging counts too, so don't just look at marts.

Then check that list against the reality you saw in the lineage. Which of your public models is your biggest consumer actually using today, and which ones are public but sitting completely unused?

Does access explain the pattern you found? Is everything mediapulse_analytics actually touches public, and is everything it doesn't touch either not public, or just genuinely unused despite being available? If access alone doesn't fully explain what's used and what isn't, what does?

Hint: where the real cross-project refs live

mediapulse_analytics/models/marts/content/fct_content_performance.sql and .../revenue/fct_ad_revenue.sql are the only two models that currently cross the project boundary. Open both and note every mediapulse_base model each one touches.

3. Read your own documentation on fct_streaming_events

fct_streaming_events sits in the streaming marts folder. From step 2 you already know marts default to access: protected in this project, and this one's no exception - so mediapulse_analytics can't ref() it at all yet, regardless of whether it's actually ready. Open its description in mediapulse_base/models/marts/streaming/_marts__streaming__models.yml and read exactly what it admits about itself anyway.

Then check mediapulse_base/README.md for the rest of the story: there's a section that tells you, in plain terms, why this model isn't ready to build on even setting access aside, and what your own team has already committed to about it.

Write down: what's actually wrong with fct_streaming_events today, and why - even once you do decide it's ready and flip it to public - access alone was never going to be what made it safe for a consumer to depend on.

4. Plan how you're going to fix that

You already know the goal: strengthen this project, and hand mediapulse_analytics a fct_streaming_events that's genuinely safe to depend on. You're not writing any code yet - and that includes not touching access yet either: flipping a protected model to public should be the last step of this plan, not the first. Sketch a plan that covers, at minimum:

  • What has to actually change in the model before the issue you found in step 3 is gone, and what test would prove it's gone rather than just hidden?
  • Beyond fixing the model itself, what would you want to be true about your own production job before you'd call this "ready"?
  • Once it's genuinely ready, what would you want locked down (docs, tests, maybe a contract) before you flip fct_streaming_events from access: protected to access: public - and how would you actually signal to mediapulse_analytics that it's now safe to build on?
  • Right now, every model mediapulse_analytics actually references from you is staging - the only folder that's public today - meaning they're rebuilding your business logic themselves instead of consuming it. If they want to eventually replicate what your finished production dataset actually looks like, is opening up more staging models the right fix, or would you rather open up specifically the marts that are genuinely ready, one at a time, and leave the rest protected?

You'll act on this plan for real in Advanced testing and dbt Mesh. This is just you getting there with a plan instead of scrambling at the last minute.

Deliverable: your ranked list of gaps (1), your public-vs-actually-used table (2), your diagnosis of what's really wrong with fct_streaming_events and why access alone was never going to cover it (3), and your fix-it-then-open-it-up plan (4).

Done?

You've mapped exactly what your project exposes to other teams, found the gap between what's public and what's actually used, and diagnosed, in your own words and from your own docs, why the model your biggest consumer wants most isn't ready yet.