Common Data Pipeline Patterns You’ll See in the Real World
A practical look at the many ways data pipelines show up inside real companies
Hi, fellow future and current Data Leaders; Ben here 👋
This is the first newsletter for 2026!
One of my goals in 2026 is to put together series. So this is the first of a longer series focused on data pipelines. I wanted to start out by discussing the types of data pipelines I’ve seen in terms of how they are used as data pipelines can be used for more that one specific use case.
Before we jump in to talking about backfills, I wanted to share a bit about Estuary, a platform I’ve used to help make clients’ data workflows easier and am an adviser for. Estuary helps teams easily move data in real-time or on a schedule, from databases and SaaS apps to data lakes and warehouses, empowering data leaders to focus on strategy and impact rather than getting bogged down by infrastructure challenges. If you want to simplify your data workflows, check them out today.
Now let’s jump into the article!
Whether you’re working at a large enterprise or a small business, there has likely been some need to take data out of the various source systems, process it, and then use it for either operational or analytical purposes.
Add in a few lines of code or a low-code solution, and the term data pipeline might start getting thrown around.
This might make some data engineers angry, but if you think about it, someone extracting data from a data source into Excel, adding in VLOOKUPs, some data cleansing via formulas and IFELSE() statements is essentially building a data pipeline….
Ok, it’s not the exact same thing, but when you stop and think about it, it can functionally solve a similar problem(although often in a more limited and specific way)
My point is that there are a lot of different ways and reasons people build data pipelines.
So, to kick off 2026, I wanted to discuss some of the key reasons data pipelines exist and the types of pipelines you will run into.
Source Standardization Pipelines
Some of the first pipelines I helped build and manage were focused on taking data sets from dozens of companies and standardizing them to a single core data model. In particular, this involved getting data via SFTP in different formats, including comma-delimited, pipe-delimited, XML, and even positional files, where you had to have a separate file that would define which columns contained which rows.
This might be unfamiliar to data engineers who are accustomed to build data pipelines to answer questions around SaaS products such as retention and churn.
But this is a problem I’ve run across now many times across many different industries from health care to retail and real estate to name a few.
In many cases this wasn’t even purely for analytics. The centralization and standardization of the various data sets allowed the companies to provide operational benefits or other services. For example, maybe you’re trying to create a marketplace and need to centralize dozens of different inventory sources.
The challenge when building these data pipelines is usually that amount of effort required to onboard and actually create scripts that can manage all the variations of how different data will come in. This is referred to as mapping.
You’ll need to:
Standardize values such as gender which can often come in as a number, single letter or the written out word
Standardize on categories, I’ve seen this a lot in retail where products might be in the same category, but one might use an abbreviation or a different word that means a similar thing
Fix date and format inconsistencies, such as different time zones, different date formats, or missing values entirely
And of course more such as how each data set might be appended. You can mitigate some of this by asking your external partners to send data in a way that is standardized, but it’s difficult to fix every issue.
Once you do have a standardized data set, you can build multiple products off this data. Whether it be a marketplace or an industry level report and because it’s all standardized it’s easy to apply new products and features for all your customers.
The one final point I will add is that this is not just limited to SFTP data sets, I’ve worked with companies that pull in data from APIs as well.
