smaple.tr
AI & Data

The Modern Data Stack: Layers, Tools, Costs and What to Build First

Mehmet Kurtipek
May 6, 2026
14 min read
AI & Data
Modern data stack
dbt
BigQuery
Data engineering

A modern data stack is a set of separate, cloud-hosted tools that copy data out of the systems a business runs on, load it into a cloud warehouse, transform it there with SQL and serve it to dashboards and other tools. A typical line-up is Fivetran or Airbyte for loading, BigQuery, Snowflake or Databricks as the warehouse, dbt for transformations, a scheduler, and a BI tool such as Metabase or Power BI. For a small team the software can cost little or nothing at current list prices; the real cost is the time someone spends owning it.

The name has aged faster than the idea. It became a buzzword during the 2020 to 2022 funding boom, and in February 2024 Tristan Handy, whose company makes dbt, argued that it had stopped meaning anything and said he would just talk about "analytics" from then on (his post). The vendors have since consolidated. Fivetran bought Census (reverse ETL) and Tobiko Data (the company behind SQLMesh) in 2025, then completed its merger with dbt Labs on 1 June 2026 (press release). So treat "modern data stack" as the name of an architecture. Fewer companies sell the pieces now, and the architecture is still a sensible default for analytics.

The layers and what each one does

Layer

What it does

Common tools (October 2026)

Sources

Where the data is created

Your app's database, Stripe, HubSpot, Salesforce, Google Ads, spreadsheets

Ingestion

Copies source data into the warehouse, raw, on a schedule

Fivetran, Airbyte, custom Python, change data capture

Warehouse

Stores everything and runs SQL over it

BigQuery, Snowflake, Databricks, Redshift; PostgreSQL or DuckDB when the data is small

Transformation

Turns raw tables into clean, tested, documented models

dbt, SQLMesh

Orchestration

Runs the steps in order and raises an alarm when one fails

A scheduled CI job or cron at first; Airflow, Dagster or Prefect later

BI and activation

Dashboards, shared metric definitions, and sending data back to business tools

Metabase, Power BI, Data Studio, Looker; reverse ETL such as Fivetran Activations

The design choice underneath all of it is ELT: load raw data first and transform it inside the warehouse, so the history is still there when a business rule changes. Our ETL vs ELT post covers that decision on its own, including incremental models and how to keep personal data out of the raw layer.

Ingestion: buy the common connectors, write the odd ones

Connectors for well-known SaaS APIs are repetitive work: authentication, pagination, rate limits, schema changes. Paying for them is usually cheaper than having an engineer maintain them. Fivetran lists 700+ managed connectors and bills by monthly active rows (MAR), counted separately for each connection. A row counts once a month however many times it changes, so the bill depends on how many distinct rows change; how often each one changes doesn't matter. The Free plan covers 500,000 MAR a month. On the paid plans, each standard connection with between 1 and 1 million MAR also carries a $5 base charge on top of usage, and Fivetran's pricing page puts a Marketo connection with 847,574 MAR at $424.54 a month on the Standard plan (Fivetran pricing). Airbyte is the main alternative: an open-source edition you host yourself, or a cloud plan from $20 a month.

Write your own extraction when there is no connector, or when the source is too unusual for one. Scraped websites are the extreme case. On the odds platform we built and ran for EdgeHunters, each of 44 sportsbooks had its own spider, and the books changed their sites without warning. To keep the feed usable, validation and anomaly checks posted alerts to Slack, and a third-party odds API took over automatically when a scraper failed (case study). If you write extractors yourself, build both of those in from the start: a check after each run that the columns and row counts look normal, and a decision, made in advance, about what happens when a source stops sending data.

Warehouse: match your cloud and your size

BigQuery is serverless, so there is no cluster to size. On-demand queries cost $6.25 per TiB scanned after the first free TiB each month, and the first 10 GiB of storage is free too (BigQuery pricing). That makes it a cheap place for a small team to start, as long as someone partitions the large tables and keeps dashboards from scanning them in full: a query that reads every column of a 500 GiB table costs about $3 at the on-demand rate. Snowflake runs on AWS, Azure and Google Cloud and lets you give each workload its own virtual warehouse, billed per second with a 60-second minimum every time it starts. Databricks fits teams whose data also feeds Spark jobs or machine learning, and Redshift fits teams that already run everything on AWS.

When the data is small you may not need a cloud warehouse at all. If your largest table fits comfortably on one server, a PostgreSQL read replica or DuckDB can run the same dbt models; dbt Labs maintains the PostgreSQL adapter, though so far only for dbt v1, and DuckDB works locally on both v1 and v2. Moving to a cloud warehouse later means fixing some SQL dialect differences; the logic stays the same. If object storage and open table formats such as Iceberg are on the table, our data lake vs data warehouse comparison goes through that choice. Self-hosting also covers data residency: if data has to stay in a country where your warehouse vendor has no region, a PostgreSQL or ClickHouse server there, with dbt on top, keeps the same layers without a cross-border transfer.

Transformation: dbt, in layers

dbt keeps each transformation as a SQL select in its own file, works out the order to run them in, and runs tests and builds documentation from the same project. Most projects use three layers:

  1. Staging: one model per source table that renames columns, fixes types and removes duplicates.
  2. Intermediate: joins and logic that several outputs share.
  3. Marts: the fact and dimension tables people query, such as orders, subscriptions and customers, in the dimensional style Ralph Kimball described.

Databricks calls the same layering bronze, silver and gold. SQLMesh is the main alternative, and since Fivetran bought Tobiko Data, both tools belong to the same company. dbt itself has been rewritten in Rust: dbt v2 became generally available in September 2026 and is what a new install gives you, while the Python 1.x line keeps getting patches (dbt Labs on the two v2 distributions). The FAQ below covers which one to pick.

Two checks belong in the project from the first day: tests on primary keys, and a freshness rule on every source. Without the freshness rule, a connector that quietly stops loading leaves yesterday's numbers on the dashboard and nobody is told. In dbt 1.10 and later, v2 included, a source file such as models/staging/_sources.yml looks like this (dbt docs on freshness):

sources:
  - name: stripe
    schema: stripe            # the schema Fivetran loads into
    config:
      loaded_at_field: _fivetran_synced
      freshness:
        warn_after: {count: 12, period: hour}
        error_after: {count: 24, period: hour}
    tables:
      - name: invoice
        columns:
          - name: id
            data_tests:
              - unique
              - not_null

Run dbt source freshness before dbt build in the scheduled job; on v2, dbt freshness does the same and also checks models that have a freshness rule. Set the thresholds from how often the table really changes. Fivetran only rewrites _fivetran_synced on rows that changed, so a small company's invoice table can go a quiet weekend without a new row, and a 24-hour error_after will then fail on Sunday while the connector is fine. Older dbt versions put freshness and loaded_at_field directly on the source instead of under config. Tests and freshness are the minimum; our data quality post covers ownership, monitoring and what to do when a check fails.

Orchestration: start with a scheduled job

With one loader and one dbt project you don't need Airflow. Fivetran or Airbyte run their own syncs; all that's left is to start dbt source freshness and dbt build after the nightly sync usually finishes, and the freshness check tells you on the nights it didn't. A scheduled GitHub Actions workflow is enough for that. Know its limits: scheduled runs can be delayed when GitHub is busy (the start of every hour is a busy time, so pick a minute such as :17), the shortest interval is five minutes, and in a public repository the schedule switches itself off after 60 days without activity. Failure notifications for a scheduled run go to whoever last edited its cron line, which is worth checking before that person leaves. A nightly five-minute run uses about 150 of the 2,000 minutes a month that GitHub Free includes for private repositories.

An orchestrator such as Airflow or Dagster earns its keep when you run many pipelines that differ from each other, need retries or have to wait for one another. On the data-migration platform we built for Tribute Technology, each kind of legacy source (MSSQL backups, MySQL databases, CSV and JSON/XML exports) had its own adapter running as an Airflow DAG. The DAGs landed raw data in MongoDB, where staff mapped the fields and approved the results before export.

BI, metric definitions and reverse ETL

Start with a cheap BI tool and replace it when you outgrow it. Metabase is free to self-host, and its Starter plan is $100 a month for five users plus $6 for each extra user. Google's Data Studio (called Looker Studio from 2022 until April 2026) is free, with a Pro tier at $9 per user per project per month. Power BI Pro is $14 per user per month, billed yearly. Looker is the enterprise option, sold by quote.

Definitions are harder than charts. If "active customer" is calculated separately in three dashboards, you will eventually see three different numbers. Define it once, in a dbt mart or in a semantic layer: dbt's MetricFlow engine has been open source under Apache 2.0 since October 2025, and LookML does the same job inside Looker. This matters more now that people point AI assistants at their warehouse, because an assistant writing SQL against raw tables will pick its own definition unless you give it one.

Reverse ETL sends modelled data back into the tools people work in, such as a CRM or an ad platform. Add it last. Census, which Fivetran bought in 2025, is now sold as Fivetran Activations.

What a small setup costs

Take a B2B software company with four sources (its PostgreSQL app database, Stripe, HubSpot and Google Ads), about 300,000 rows changing a month, 20 GB of data and five people reading dashboards. At list prices checked in October 2026:

Layer

Choice

Monthly cost

Ingestion

Fivetran Free plan, while under 500,000 MAR

$0

Warehouse

BigQuery on-demand

Well under $1 for storage; $0 for queries while they stay under 1 TiB

Transformation

dbt (v2), run in the same nightly job

$0

Orchestration

Nightly GitHub Actions workflow

$0 within the included minutes

BI

Metabase Starter, five users

$100

Total

About $100

Self-host Metabase on a small server and the software bill drops to the cost of that server; with Data Studio it drops to almost nothing. Three things push it up, and each is a meter worth watching every month:

  • Changed rows. Past 500,000 MAR you leave the Free plan and every connection is billed on its own volume, as the Marketo example above shows.
  • Bytes scanned. Unpartitioned tables and dashboards that refresh often are what turn the free terabyte into a real BigQuery bill.
  • Seats. BI pricing is per user, so 50 Power BI Pro users cost $700 a month.

None of this includes the person who keeps it running: someone has to own the connectors, review model changes and find out why a number looks wrong. That work doesn't stop after the setup, and at these prices it costs more than all the software put together.

When you don't need one

If you have one source and a handful of reports, connect a BI tool to a read replica of your database and stop there. A warehouse, connectors and dbt add moving parts that a single PostgreSQL database doesn't need.

If the data drives the product in near real time, the stack is the wrong shape. On the EdgeHunters platform, odds went from the spiders through Celery workers and Redis queues, and every five minutes the latest odds were packaged and pushed to S3 and PostgreSQL for the product to read. In 2025 we designed its next step as a streaming, event-driven pairing architecture to cut latency further. A rule of thumb: if people read the numbers once a morning, a nightly batch is enough; if a number is worthless after a few minutes, you are building an application pipeline, and a warehouse with a nightly dbt run won't serve it.

The order to build it in

  1. Pick one question someone asks every week and can't get answered quickly today, such as monthly recurring revenue by plan. Write down who asks it and where the number comes from now.
  2. Load only the sources that question needs, raw.
  3. Write staging models with key tests and freshness rules.
  4. Build one mart and one dashboard, and reconcile the headline number against a figure the source system produces independently before anyone relies on it.
  5. Schedule the job and send failures somewhere a person reads them, at weekends too.
  6. Move on to the next question. Add Airflow, a semantic layer or reverse ETL only when a concrete need shows up.

Step 4 is the easiest one to skip. On Savings United's BI team, where we worked from 2020 to 2024, payout data from affiliate networks such as CJ, Awin and Impact had to be reconciled with the warehouse before reports could be trusted, so we automated QA that compared the two (case study). That stack wasn't the textbook one: SQL Server, Python ETL jobs and Looker, with the cost fact table moving to BigQuery in 2024, a migration we validated. The stages were the same, though. We documented the Tradedoubler payout pipeline from raw data through staging and the warehouse to Looker, and the warehouse had a weekend on-call rotation that we shared.

Frequently asked questions

How is it different from a traditional data warehouse?

A traditional setup ran on servers you owned, transformed data in an ETL tool such as Informatica or SSIS before loading it, and had fixed capacity. A modern stack uses a cloud warehouse that scales storage and compute separately, transforms after loading, and is assembled from tools you can swap one at a time.

Is dbt free?

Running it yourself costs nothing. dbt v2 comes as two distributions: dbt, the default, under a dbt Labs licence that is free to use, and dbt-oss, the Apache 2.0 open-source subset. The Python dbt Core 1.x is Apache 2.0 as well. The hosted dbt platform is free for one developer seat and $100 per seat per month on the Starter plan; Enterprise is priced by quote.

Should I start on dbt v1 or v2?

Check your warehouse first. On v2, BigQuery, Snowflake, Databricks, Redshift and DuckDB are generally available, ClickHouse is in private beta and PostgreSQL isn't supported yet, so a PostgreSQL project stays on 1.x for now (adapter status). Otherwise start on v2: new installs default to it and it parses large projects much faster. dbt Labs says 1.13 will be the last 1.x release with new features and will get security patches for years, so an existing v1 project isn't stranded. The source file above parses and runs on both 1.12 and v2.

If you want a hand choosing or building the first version, that's part of our data and AI work.

Related Articles

August 11, 2026

MLOps Guide: What to Set Up First, and What Can Wait

MLOps gets a model into production and keeps it useful there. Here is the order we'd set it up in, with current MLflow and DVC examples, and what changes for LLM features.

Read More: MLOps Guide: What to Set Up First, and What Can Wait
August 9, 2026

Computer Vision Applications: Where It Works and Where It Struggles

Where computer vision works today, from inspection lines to invoices and medical scans, what makes some applications hard, and when a general vision model saves you training your own.

Read More: Computer Vision Applications: Where It Works and Where It Struggles
August 4, 2026

AI Agent Development: When You Need One and How to Build It

An AI agent is a language model working in a loop with tools. When you need one, the loop in Python, choosing a framework, keeping it safe, and what each run costs.

Read More: AI Agent Development: When You Need One and How to Build It