What It Actually Takes to Build a Data Pipeline System
A breakdown of the components, tradeoffs, and realities of building your own data pipeline system
Hi, fellow future and current Data Leaders; Ben here 👋
Today I am continuing my series on data pipelines. In the prior article we discussed the types of data pipelines that exist. Today, we’ll be discussing the components you’ll need if you plan to build your own data pipeline from scratch.
But before we jump in, I wanted to share a bit about Estuary, a platform I’ve used to help make clients’ data workflows easier and am an adviser for. Estuary helps teams easily move data in real-time or on a schedule, from databases and SaaS apps to data lakes and warehouses, empowering data leaders to focus on strategy and impact rather than getting bogged down by infrastructure challenges. If you want to simplify your data workflows, check them out today.
Now let’s jump into the article!
When I first started in the data world, it was common that many data teams would build their own data pipeline solutions. There were still dozens of options in terms of off the shelf tools of course, nevertheless, you’d see custom pipelines developed everywhere.
In 2025, I saw less of this.
In fact, in many cases data teams would go straight to picking tools or solutions.
But let’s say you do want to go down this route. You want to build your own data pipeline solution?
How would you do it?
What Components You’ll Need
Below I’ll outline the components most every data pipeline system I’ve worked with requires/has had.
Secrets And Connection Management
I am going to start the list of components off with secrets and connection management.
Because this is how you’ll likely set up source and destinations, without sources and destinations, you really have no reason to build your pipeline.
You just have orphaned SQL logic doing nothing, and Python pushing data nowhere.
It’s also crucial in how easy you make it to manage the rest of your system.
Do you want your data team members to have to write a custom connection script every time?
If a source or password changes, are you making it easy to update the information in a single place or multiple places?
Do you make it easy to store securely without exposing it to the repo?
Small details here matter and add up over time. If I have to make a separate connection reference every time I need a new table from the same database, that’ll be terrible.
And for those out there who assume they’ll only ever need to connect to a few sources, you’d better hope you’re right.
Logging And Monitoring
When you build your first pipeline system, maybe you put in a few print statements to track where your pipeline has succeeded and failed. As you start building a more generic system, logging needs to be included.
You need to be able to trace back and figure out if there was an issue with an external library, with a specific module inside your data pipeline system, or an actual problem with a pipeline you’ve written.
Think “we can’t find this library” vs “we can’t find this table”.
You also want to know on which run this occurred. Was it data from a specific date, or if you think in terms of Airflow, one of those little red boxes?
Without logging, debugging is impossible, and as more code gets generated by AI, we are going to need even more specific error messages and traceability in order to figure out what we need to fix.
Dependency Awareness(Graphs)
Not everything has to be DAGs, but you’re going to need some sort of dependency awareness. I recall building a very naive version of this at one of my first jobs, where I simply created a table that kept track of which jobs had run, on what date, and the numeric step in the process.
This quickly falls apart if your pipeline needs to change frequently or you need to build anything with a smidge of complexity. Then you start needing to look at solutions like Airflow and dbt, and how they handle referencing dependencies.
For example:
extract_orders.set_downstream(transform_orders)SELECT * FROM {{ ref(’stg_orders’) }}
Some how, you do need to tell pipelines what the prior task or tasks they need to wait on are so they can check the status.
Execution Engine Routers
I’ve seen multiple companies now spend much of their data teams’ budget and time simply on migrating data from Databricks to Snowflake.
Why?
Because they use Databricks to run the expensive, heavy early data processing and then use Snowflake as a service layer.
Data teams want to be able to pick the compute they need, and so this is somewhat of a newer concept for data pipeline solutions. We now have multiple compute engines that people want to use when processing data. Think DuckDB, Presto, and maybe just a local instance of Spark.
Some engines are cheaper or faster, and still others handle larger data sets better. I foresee solutions in the future, routing more and more of this traffic to optimize for what your team is looking for in a specific pipeline. We actually had this at Facebook(although we had to tell it which engine to use)
In the same way, it’s worth considering if you ever plan to build your own pipeline solution. I wouldn’t build it right away. But if you’re looking to further optimize your own internal solution or if you’re thinking bigger and building a dbt competitor(which several people have reached out to me saying they are), then I’d consider adding in routing.
