Is It Time to Say Goodbye to Data Engineers?

The Data Engineering Dilemma

Hi, fellow future and current Data Leaders; Ben here 👋

Today, we are going to dive into an issue I’ve noticed that seems to oscillate in the data world every few years. That is the removal of data engineers. After all, we tend to get in the way and slow things down, right?

Before we dive into the article, I’m also excited to share that I’ve started to put together a community for data leaders. The goal of this community is to create a space where data leaders can learn to be more impactful, share their issues and challenges, get other data leaders to weigh in, and so much more, so sign up here to get an invite.

With that out of the way, let’s jump into it!

---

Intro

Ever since tools like SSIS came onto the scene, vendors and business leaders have been on a mission to remove what they see as the biggest roadblock to data-driven decision-making: data engineers.

Or their counterparts—DBAs, ETL Developers, and Data Architects.

Sure, not everyone says it so explicitly, but you can see it in vendor marketing and in the decisions made by the business.

I remember talking to a veteran data expert who’s been in the field for three decades. They told me that when SSIS first launched, people were genuinely afraid for their jobs. The idea that you could just drag-and-drop tasks that once required code was nerve-racking. But if you’ve used SSIS, well, you know the truth.

To some extent, I get why the idea is appealing. When a leader requests a report, a software engineer wants to modify an application table, or a data scientist wants to explore a new dataset, who’s the one slowing down the project?

The data engineers.

With their rules. Their governance. Their insistence on building robust, scalable data pipelines instead of quick fixes.

They really slow down the workflow!

So, sure, from that perspective, I understand the desire to move faster. But before you go all-in on the “let’s remove the data engineers” strategy—or if you're currently fighting a business leader who’s hellbent on it—let’s take a look at what actually happens when companies try to cut out data engineering. Spoiler alert: it rarely ends well.

What Happens When People Remove Data Engineers

I’ve walked into multiple projects where data engineers were cut from the equation, and I’ve heard of plenty of horror stories about similar situations. At first, it seems like a win. The short-term goal is moving fast.

Suddenly, data scientists, semi-technical end-users, and other SMEs are cranking out workflows at record speed. Maybe leadership invested in a low-code tool, and now reports, dashboards, and pipelines are flying out the door.

Success!

Well, like many short-term wins, this can go on for a few months, maybe even a year. But then reality sets in.

Before long, your data warehouse (Snowflake or whatever you’re using) is cluttered with an endless mess of tables, views, dashboards, and other artifacts that all tell slightly different stories. There’s no standardization, no governance, and no one treating data like a product.

Eventually, someone—probably an overwhelmed analyst or a frustrated executive—suggests bringing in more structure. More governance. More control.

And just like that, the pendulum swings in the other direction and the cycle repeats.

Don’t believe me?

Airbnb went through this where as they put it:

For several years, Airbnb did not have an official Data Engineer role. Most data engineering work was done by data scientists and software engineers who were recruited under a variety of different monikers. - Data Quality at Airbnb

This worked great at first because it was fast, but eventually they ran into issues around:

• Data quality

• Ownership

• Standards

• Governance

The truth is that, speed alone isn’t the answer to your data accessibility challenges.

A House With No Foundation

If your data strategy is built on nothing more than unstructured data swamps, CSVs dumped into Azure Blob Storage, and a tangle of Python and SQL scripts, it’s only a matter of time before it all collapses.