Today I want to talk about data stacks and layering on complexity…

Before we jump in to todays article. I wanted to let y’all know that today’s article is sponsored by me, the Seattle Data Guy!

Our team has helped dozens of companies turn data in to actual business outcomes. We’ve also helped companies set-up their data stack from the ground up as well as untangle their current data infrastructure. If you’re looking for an experienced data consulting team to help you set-up your data infrastructure and strategy, then set-up some time with me today!

Now let’s jump into the article!

Tech folk are like onions; we have layers.

Actually…its more like we love the idea of layers.

Network layers.

Medallion architecture.

Layer, after layer, after layer.

On one side, these layers help delineate where on process or job starts and another ends..

But on the other side, we tend to keep layering more and more layers on top of each other. Adding new roles, new tools, new platforms.

All to make things “easier”.

Think of the Modern Data Stack.

Snowflake, Databricks, and all the other tools. They did make things easier in so many ways. But also, I’ve helped companies save millions on compute costs.

Many times, because data teams kept adding layer upon layer that just needed to be removed.

Don’t worry, we’ll discuss that later. Let’s start by discussing the pros and cons of systems and tools that make development easy.

Our Solution Makes It Easy - How It Happens

Every generation of tools promises the same thing:

“We’ll make this easier.”

What they actually do?

Add another layer.

I just read an article discussing how Databricks is trying to position itself in that light.

All that made me think of was one of the first articles I wrote nearly a decade ago. I was talking about Tableau and how the fact that it was so easy made things dangerous.

And I can tell you from experience this is true and it’s helpful. You can test out new ideas, dig into data faster, etc.

Of course, simply pumping out more code, more dashboards, more artifacts..more layers…faster has never been the key problem. In fact, in many ways, it’s created its own set of new problems. You’ve got to deal with every type of sprawl under the sun.

• BI sprawl

• Pipeline sprawl

• Model sprawl

• Agent sprawl

• Cost sprawl

• System sprawl

The image below, doesn’t even show all of that. And it’s also starting to sprawl and in each of those boxes is it’s own complex sprawl of code, dashboard metrics, calculations, and locally optimized workflows.

These systems end up like fractals. Every box you go into has yet another set of boxes, and arrows.

There are many reasons for this.

• A department head wanted to run their own AI/Data project

• The data analysts and engineers wanted to use different tools

• No one wanted to actually make a decision, so you picked all the tools

• Outcomes were not the key focus

This can create costs that explode and systems that become harder and harder to maintain.

And no amount of data governance lathered on top of all this will make it more sane.

You’ve got a data catalog for your data catalogs because hey, why not!

You’ve got orchestrators for your orchestrators.

And you’re ingesting data from your ingestion tool about how your ingestions are running.

We’ll likely need AI not only to develop code faster but to understand what in the world is going on inside some of these spaghetti systems we are constructing.

The Devil’s Advocate - Spending More On Tech Makes Sense No?

But let me take another side to this.

On the flipside, some companies I’ve worked with didn’t have full-time data engineering teams or had far smaller ones than in the early 2020s.

Sure, in some cases, maybe they spent an extra $125k a year on Databricks and or Snowflake….

But they saved a significant amount on hiring a DE team. Based on the average data engineer salary, if you can cut 2 from your team and spend an extra $125k on Snowflake and Databricks…thats not a bad deal.

We are removing people from the layers and replacing them with technology!

Yes, there might be some random scripts calling other random scripts, but it all works right?

I actually recall one friend making a joke about this where they referenced the fact that, sure, a company could put effort into optimizing their data model, their data pipelines, etc.

Or, they could crank their Snowflake instance up a few notches, pay a little more, and fire half the data engineering team, and it would be significantly cheaper.

Some companies are essentially doing this now. Instead of paying a full-time data engineer, they hire someone to help set up their data stack and then come in every so often to simply maintain it. I know, as I’ve been that someone.

The Realities Of What This Leads To

Here are some realities I believe we will continue to see in this AI-driven data world where we seem to building yet more technical layers.

Business Logic Bloat Will Only Get Worse

When dbt became popular one thing that many data teams found is that model bloat was real. You’d keep building out more and more models now that a broader set of users that manage to obfuscate hidden business logic.

The easier it is to translate business logic into code, the more you take out of peoples brains and put it into technology. In some ways thats great. In other ways, some decisions made will be to try to get code to cover even more edge cases than we have in the past. Because, why not? Just ask your LLM to code one more, it has 0 cost to you?

People Will Still Struggle To Connect Things To The Business

I believe despite being able automate more and do more technically speaking. Teams will still run into a similar problem that we have today. At least until LLMs figure this out. That is that people will still struggle to connect business value to technical output. We gave more people access to data, yet many companies are still struggling to answer basic questions and in turn struggling to make decisions. Some of this, I believe is due team sizes and so many teams specializing in technologies vs. the business. So there is a chance, that we might be able to avoid this, but I don’t think we will(at least not for a while)

The Fractals Will Grow

In theory, I believe we should take some of this tooling, LLMs, etc and in turn start removing layers of tech to simplify tech stacks. In reality, this is not what I see. I love that pointed out the 37 tool data stacks from the 2020s. He might even be under counting.

Data stacks weren’t just ETL, data warehouse, data visualization and data catalog. I’ve come across, ETL → Data Lakehouse → another ETL tool and orchestration tool → dta warehouse → yet another ELT → database →semantic layer → six different BI tools. Not to mention some custom code in between!

I don’t think we’ll be getting rid of that. I think most companies will just layer AI on top of all of that.

We Are Growing The Pie Of Automated Use Cases

I do think will grow the pie in terms of how easy it is to build automated use cases. Whether the results are better is hard to say. I lean towards the side of, most people will just add complexity for complexities sake because that’s what I’ve seen in the prior few hype cycles. Some tooling expands the usage of a popular idea.

• Machine learning becomes popular - Here are some Python libraries you can use to implement models without understanding them. You’ll still struggle to find good places to implement them as well as know how to properly deploy them.

• Big Data becomes all the rage - Here are some tools that makes handling big data easier. Companies will still struggle to become data-driven and dashboards still take too long to load.

I am sure you could find even more examples there. Technology makes a lot of things easier but it’s still challenging to connect good ideas with outcomes.

Tactical Take Away

Before adding another layer, ask yourself three questions:

1. What problem does this layer actually solve? - I find many engineers become a “Snowflake” Data Engineer or “Databricks” Data Engineer, which sure, that’s great. But, you eventually get so divorced from outcomes that you can easily end up creating layers and work for the sake of it.

2. What happens if we don’t add it? - More often than not, the answer is “nothing breaks.” and “We actually now need to hire someone to take care of this layer”.

3. Who owns it six months from now? - Because every layer eventually becomes someone’s problem.

If you can’t answer all three clearly, you’re not building leverage.

You’re building liability.

The goal isn’t fewer tools for the sake of it but nor is it more for mores sake.

The goal is a system you can actually understand, maintain, and tie back to the business.

And that takes more effort than just signing another contract for a new tool or building another ten dashboards.

As always, thanks for reading! I hope to see you in the next one.

Articles Worth Reading

There are thousands of new articles posted daily all over the web! I have spent a lot of time sifting through some of these articles as well as TechCrunch and companies tech blog and wanted to share some of my favorites!

From “Vibe Coding” to Guided Coding

Enterprise engineering teams face a paradox: while generic AI tools increase individual speed, they often introduce unpredictability, security risks, and technical debt at scale.

This white paper details a pilot program conducted by Brainly in partnership with Codestrap. By deploying Larry AI, an agent powered by X-Reason™ technology, Brainly successfully demonstrated that guided workflows can outperform generic AI tools in reliability, cost, and safety.

The pilot focused on “LLMifying” Brainly’s complex Data Access Object (DAO) layer and building custom Larry AI workflow for this area of the codebase. The results were decisive: Larry AI achieved 90% one-shot correctness , reduced inference costs by ~96%, and-most critically-enabled mid-level engineers to safely ship code that previously required deep domain expertise.

Read More Here

Speed Without Understanding - One of the Biggest Risks in Data Engineering

“I just nuked all our dashboards.”

That’s the title of a heavily discussed post on the data engineering subreddit(Post and link included at the bottom of this section). When I first read it, I assumed the author was going to talk about deleting all their dashboards, no one noticing, and then wondering if their job even mattered.

That wasn’t the case.

Read More Here

End Of Day 212

Thanks for checking out our community. We put out 4-5 Newsletters a month discussing data, tech, and start-ups.

If you enjoyed it, consider liking, sharing and helping this newsletter grow.

Share

One thing most of us data engineers dislike are backfills. Why is that? And what does backfilling require?

Before we jump in to talking about backfills, I wanted to share a bit about Estuary, a platform I’ve used to help make clients’ data workflows easier and am an adviser for. Estuary helps teams easily move data in real-time or on a schedule, from databases and SaaS apps to data lakes and warehouses, empowering data leaders to focus on strategy and impact rather than getting bogged down by infrastructure challenges. If you want to simplify your data workflows, check them out today.

Now let’s jump into the article!

At some point, if you work in data, whether you’re an analyst or a data engineer.

You’re going to have to do it.

You’re going to have to backfill a table.

Actually, it’ll probably be pretty early in your career. Backfilling or rerunning a pipeline is just a necessity, AI or not.

There are plenty of reasons why you might need to backfill a table..sadly.

Talking to data engineers…many of them dislike the process of backfilling.

So let’s start there. Let’s discuss why we backfill as data teams and why we dislike it so much.

Why Do We Need to Backfill Data?

Backfills exist for many reasons. For example, systems are not static, and people make mistakes. In turn, tables and data sets need to be rerun.

Some of the most common reasons you’ll need to backfill include:

1. Late or corrected source data - Upstream systems change historical records all the time. Maybe the data was being recorded incorrectly, or if you’re getting SFTP files, they might have been sending you bad files that no one caught. Now you’re going to have to backfill at least a specific date or customer cut of that data.

2. Bugs in your data pipelines - A common reason why tables need to be backfilled is that there was a bug. Something was wrong, and maybe just running an update statement isn’t sufficient. So now you’ve got to rebuild the table with the new logic without disrupting end-users.

3. Schema or logic changes - This was a common reason I needed to backfill at Facebook. When columns were removed at Facebook or, in some cases, certain data type conversions were required, we’d have to rebuild the table. Then add in new logic changes or sources for data, and you’ll likely need to reload the entire table.

These are, of course, only a few reasons!

Why Data Engineers Dislike Backfilling

If you ask a data engineer what they don’t like doing, I am sure backfilling will be one of the few things they reference, besides migrations.

Here are a few reasons why.

• Scale- In some cases, backfills mean waiting hours, if not a day or two, to rerun a table. At Facebook, sometimes backfilling a table meant needing to rerun thousands of jobs for each partition and each of the upstream tasks. That means there are a lot of ways things can go wrong.

• Cost - You’d better be sure your backfill updates are right. Having to rerun a backfill job on pay-as-you-go technology will be expensive, especially if you have to load the data from raw.

• Time Consuming - There are multiple ways backfills take time. They can bump into daily jobs, especially if you are on-prem. They also take time out of the day of an engineer who has to ensure all the data is accurate and runs as expected. It’s just one giant time suck that keeps the data team from delivering new work.

• Blast Radius - So you’ve built a table that everyone at your company uses and relies on. Great, now it’s going to take even longer to backfill. I had multiple cases where I’d be backfilling a table that had dozens, if not hundreds, of end-users. You’re going to need to update them and make sure they know what’s happening and if they need to do anything. If they have to modify their pipelines, then that further drags out your process.

• Trust – If stakeholders see numbers change unexpectedly, they are going to question it. So one of the many reasons data teams dislike backfills, especially if they have to do it frequently, is that it can erode trust in the overall dataset.

Backfilling “Controversy”

As I was going through to see what other people had said about backfilling over the past few years, I ran into a discussion on a post between and from over a year ago(not trying to restart a fight here, just making sure people don’t feel like I am talking about them behind their backs).

The argument itself became a little heated, but stripping away that component I do think there is something worth talking about.

I think both Brian and Zach have different experiences in different systems that, in turn, have different requirements for backfilling.

A goal you should have when backfilling(amongst the obvious of backfilling) is not to run a bunch of random SQL scripts against production.

Instead, you should create an approach that lets you maintain a repeatable process that balances making changes safely with giving space to ensure the new data is correct.

Read more

Today will be getting back into my series on data pipelines. One question that I believe is important to answer is why? Why even build data pipelines?

But before we jump in, I wanted to share a bit about Estuary, a platform I’ve used to help make clients’ data workflows easier and am an adviser for. Estuary helps teams easily move data in real-time or on a schedule, from databases and SaaS apps to data lakes and warehouses, empowering data leaders to focus on strategy and impact rather than getting bogged down by infrastructure challenges. If you want to simplify your data workflows, check them out today.

Now let’s jump into the article!

When I first started in the data world, no one around me used the term data pipeline.

I heard terms like integrations, automations and ETL.

In fact, I am not even sure when I first came across the term. But if you’re a data engineer in this modern era, then much of your time is spent, building, maintaining and keeping data pipelines running smooth.

Even with AI, you’re probably still finding yourself opening up 3,000 line queries, and the occasional custom data pipeline system.

What a Data Pipeline Actually Does

When you look at data pipelines, here is likely what people might say they do.

1. Move data from a source to a destination

2. Sometimes they transform that data

3. And they do all of this repeatedly and reliably without human intervention

That’s the technical function of a data pipeline.

How it happens can vary.

This could be automated SQL, Python scripts, Airflow, Estuary, SSIS, Glue, and so many other tools.

But you do need to think beyond just this when it comes to data pipelines.

Pulling in a recent post from .

It’s important to think beyond just moving data from A to B. And start thinking in. outcomes and ownership.

What is the data pipeline actually doing?

The Real Reason Data Pipelines Exist: Trust

We alluded to this above, but let’s talk about why data pipelines exist. Because, hey we could just manually load data into databases.

Just use:

COPY INTO analytics.raw_orders

FROM @raw_stage/orders/

FILE_FORMAT = (TYPE = ‘CSV’ SKIP_HEADER = 1)

ON_ERROR = ‘CONTINUE’;

Done!

No need to automate anything, right?

After all, we are just moving data from point A to B.

Well, there are many reasons we automate data workflows and turn them into data pipelines. Here are the key benefits we get.

• Timeliness

• Accuracy

• Consistency

• Recoverability

• Scalability

But it goes even beyond just recoverability and consistency. We really are trying to make data more valuable. In order to do so, we must also consider these pillars.

• Integration

• Availability

• Outcomes

Now I will say, not every data pipeline you build will have the goals listed above. Especially the ones listed in other benefits.

In some cases, data pipelines are merely integrated data from a CRM to another internal system other than reporting.

Still others might pull data out of your CRM, run some calculations, and reinsert it.

I am merely saying this to point out that not all data pipelines are used to push data into data warehouses. There are plenty of other reasons a data pipeline might exist.

Why You Need To Care About These

Integration

Many data teams aren’t building data warehouses; they are just replicating their databases and CRMs into Snowflake or Databricks. Just isolated siloed data that was once in separate systems, now in their own schemas and un-integrated data sets.

Part of what the data pipeline is supposed to handle in terms of logic(and as determined by the data modeling process) is the integrations. The parsing, cleaning, and adding of keys that allow you to join data across systems. This also means you’ll likely need to consider what data sets will need to join with each other in the source systems themselves.

Availability And Usability

Many data workflows require data analysts go to the source systems and extract the data in an Excel, then from there they will need to manually process, set-up VLookups and build out a “database” in Excel.

Part of what data pipelines do is move data into the data warehouse making said data more easier to access.

And this is not just for end-users like analysts, but also automations and LLMs. Having data centralized means it’s easier to work with said data, especially when it’s well modeled.

As data becomes easier to access by the right users, the more they can actually use it for.

Scalability

At a certain point, having half automated scripts run by cron might be too chaotic. Sure, if you only need 2-3 simple data workflows managed. This might be fine.

But as your data use cases grow.

As your data team grows.

As the end-users of said data grows.

You’ll want data pipeline systems that make it easy to automate.

Think about having to rerun 200 data pipelines. That’s logistically difficult if you can’t easily kick all the jobs off and track their successes or failures.

Outcomes

Data pipelines can be easily built without pipelines. But I think it’s important to think about the “so what”. Why are you building your data pipeline?

Is it to automate a process, and if so, does it need to ingest data into the data warehouse?

What business goal are you hoping to drive with the building of your pipeline? Every new data pipeline you build without a clear purpose just becomes a technical liability over time. It increases cost, maintenance time, etc.

So what is your team hoping to do with the data pipeline?

Here are a few examples, you could say your data pipeline:

• Reduces unnecessary discounting by analyzing win/loss data and discounts to show where deals close without price concessions.

• Improves onboarding success by identifying which onboarding steps and early product behaviors correlate with long-term retention.

• Reduces support costs by linking support tickets to product events to eliminate the root causes driving repeat issues.

• Increases retention through proactive customer success by alerting CS teams when usage drops or support volume spikes.

Timeliness

One of the great things about data pipelines is that they are easy to track and can run whenever you need them to.

Meaning, if you need them to prepare a data set prior to 8 AM, they can do that. You know how long it’ll take(assuming nothing goes wrong, and even then, likely you can set up some level of recoverability).

An analyst doesn’t have to wake-up early to make sure the data gets processed in an Excel file. Instead, it can land in a table and be picked up as needed.

Accuracy

We in the data world love talking about data quality. Well, the data pipeline is one of the many places where data can be transformed improperly. Data can be duplicated, removed, and or altered in such a way that it is no longer accurate.

Data pipelines are a great place to check for data issues.

This can occur before even processing the data to check that the source contains the expected fields and ranges of data. From there, as you transform the data throughout your pipeline, you’ll likely need to implement other checks.

Consistency

The problem with “Excel data pipelines” is they offer room for errors. You copy and paste the wrong data set or forget to update a formula.

A programmed data pipeline, is repeatable and consistent.

You can create logic to check for errors like the wrong data being inserted or if dimensional data is missing. So even if you do have some issue, it can flag it early. It also helps avoid a fat-finger issue where someone accidentally thumbs in a number.

Recoverability

Sometimes, the wrong data enters a data workflow. We want to be able to detect that and then be able to rerun our data processes easily without having to worry about what else could go wrong.

We don’t want to worry about duplicate data.

We don’t want to worry about a small step being missed.

So having the process codified ensures we know exactly what will happen in terms of data tables being populated.

Final Thoughts And What Is Coming

Data pipelines are everywhere in companies. They take many different shapes and forms, but overall, their goal is to do more than just move data from point A to B.

In case you missed my last article, we’ve already covered some of the various data pipelines that exist(I even labeled one the Excel Data Pipeline in the past).

In the next few articles, I’ll be covering several other key data pipeline topics:

1. Backfills: The Thing Everyone Avoids Until It’s Too Late

2. Building your first data pipeline, from Excel to Airflow

3. Incremental vs Full Refresh Pipelines

4. Daily Tasks With Data Pipelines - Data Quality Checks And The Problem With Noisy Checks

Articles Worth Reading

There are thousands of new articles posted daily all over the web! I have spent a lot of time sifting through some of these articles as well as TechCrunch and companies tech blog and wanted to share some of my favorites!

When Did “Rock” Become “Classic Rock”? A Statistical Analysis

By

I first grasped the strangeness of the term “classic rock” while listening to a pop-punk song. In 2004, Bowling for Soup topped the charts with “1985,” a track about a frustrated housewife nostalgically longing for her Reagan-era youth.

During the song’s bridge, our protagonist laments cultural change: “She hates time, make it stop. When did Mötley Crüe become classic rock?” It was at this moment that I—a teenager—first understood the peculiarity of genre:

1. Apparently, there was a band called Mötley Crüe.

2. This motley crew was initially classified as one genre—rock—before being reclassified as “classic rock.”

3. Contrary to my longstanding belief, “classic rock” was not etched into the Ten Commandments—it was a contemporary radio format, likely devised by someone in public relations.

Read More Here

Snowflake vs Databricks Is the Wrong Debate

Over the last few years, Databricks has been executing a strategy to take over the entire data workflow.

Maybe it never started that way.

Maybe when they first came out, they only ever planned to be a managed Spark solution. But I have a hard time believing that, mostly because I believe their leadership has the vision and capabilities to see far beyond that.

Databricks has always been pretty upfront that they want to be the end-to-end data stack. But they’ve been approaching it piece by piece.

Or should I say role by role?

Obviously, at first, their focus was on the data scientist and ML engineer.

Read More Here

End Of Day 210

Thanks for checking out our community. We put out 4-5 Newsletters a month discussing data, tech, and start-ups.

If you enjoyed it, consider liking, sharing and helping this newsletter grow.

Share

Today I am taking a pause from my data pipeline series(it’ll start again next issue) to share some thoughts about the data world and where it’s headed in the next year or so and other trends.

But before we jump in, I wanted to share a bit about Estuary, a platform I’ve used to help make clients’ data workflows easier and am an adviser for. Estuary helps teams easily move data in real-time or on a schedule, from databases and SaaS apps to data lakes and warehouses, empowering data leaders to focus on strategy and impact rather than getting bogged down by infrastructure challenges. If you want to simplify your data workflows, check them out today.

Now let’s jump into the article!

One twelfth of the year is over, at least by months, and somehow it feels like a year’s worth of events have occurred.

By the end of 2025, dozens of companies were swallowed up. Everyone wanted to buy everyone, and here we are in 2026, seeing more of that as well as some pretty slick new AI model releases.

But let’s turn towards the future, and specifically, data.

Here is what I believe we’ll see happen in the next year or two.

1) Microsoft Fabric Will Rebrand..Again

If you scroll around LinkedIn enough, you’ll likely find a few posts about Microsoft Fabric not being the “it” tool. As some people have put it:

Microsoft Fabric is Databricks from Temu

Now, it doesn’t mean it’s not growing. According to their recent earnings, Azure is growing at 39%. Y/Y.

But, if the sentiment around Fabric continues to grow in the wrong way, I’d predict Microsoft rebrands their data stack…again.

Especially with all the AI-hype. They’d be able to push the narrative that the new solution is AI-first and come up with a great new name.

Looking back over the past decade, Microsoft’s done this several times. To the point where it was a little confusing in terms of which tool is which.

So maybe they’ll try yet again.

2) 1% Of Companies Will Continue To Cry For AI While The Other 99% Are Still Trying To Export ERP Outputs To Excel

There is a lot of demand for AI. It needs to be integrated everywhere, right?

On the flip side, many companies are still sharing data via SFTP or pulling it from an API. Sure, maybe they used AI to help write the code faster.

But then they went to scroll on Instagram afterwards.

You know what AI tool I want to see(and maybe I should take a crack at it). I want to see an AI solution that lets me take an Excel spreadsheet, drop it in, and it automatically builds out a data pipeline that can replace it perfectly. Or as perfect as it can…

It’d pull out all the formulas, turn them into SQL or Python logic, and put them into a larger system of data pipelines.

Because we aren’t going to replace Excel anytime soon. Every company has it no matter what data decade they are in. Excel is capturing business logic.

So why fight it?

It’s just too easy to build a quick spreadsheet that quickly turns into a core component in a business workflow.

No one wants to fill out a restrictive form, and sure, coding is nice, but it’s also heavy.

So why not just open a spreadsheet, build it, and have an easy button to turn it into a pipeline?

Is it just that easy, right?

3) Modern Data Stacks Will Be Shaken

With all the start-ups that have been bought up recently and others raising prices I foresee a shake-up in the default approaches that companies use to build their data stacks.

On top of that I wouldn’t be surprised that we see a wave of companies needing their data stacks re-built from the ground up.

Between fragile data pipelines, changing pricing models, sunsetting solutions, and Just-in-Time data models, people will take another look at what they built and see if it actually meets their needs.

By the way, great segway, if your data team needs help revamping your data infrastructure, whether you’re on Databricks, Snowflake, Bigquery, or all of the above, then reach out for a consultation!

This opens the door for new solutions as well as, I hope, building more solid data foundations, which if you need a good way to pitch it to data leadership.

Just call it “AI-foundations”.

4) AI POCs Will Start To Build Actual Foundations

Over the last few years, we’ve been fed what endless new terms and patterns for how to make LLMs useful. I believe we’ll start seeing more crystallized patterns for how companies are actually planning to use LLMs(for more than just writing code).

Because for every fifty projects that were driven by hype, there are one or two where the engineers focused on delivering a reliable solution. When they came up against problems, they didn’t rush through them or ignore them.

They actually spent time trying to figure out how to work with what they were getting. They tried to figure out the actual value of the LLM beyond the surface-level use cases.

It takes time to develop design patterns and processes, and soon we’ll have enough iterations where some teams will be able to reliably execute ideas. Over the past few tech hype cycles the general process I see is:

1. New capability appears - A breakthrough hits (LLMs, streaming, blockchain, “big data”, etc.). Early demos look magical, but the real constraints aren’t understood yet.

2. Everyone builds the obvious thing first - For LLMs, this was:

   • Chatbots everywhere

   • “Ask your data anything” demos

   • Code generation and copilots

3. Reality sets in - Teams start to run into problems. Think hallucinations, cost blowups, security and governance concerns and in some cases, things just don’t work as expected. You need to start integrating safe-guards and best practices(that no one has created yet, you..you are the one creating them!)

4. Patterns start to crystalize - Every new capability comes with limitations. But you wont’ know them until you implement it. Until it hits scale or just has to perform an edge case you hadn’t considered.

5. Becomes a standard - Eventually new capabilities are viewed as a standard piece of infrastructure. Integrated in such a way where you notice it less because it smoothly fits into the rest of your flow.

6. The hype fades and we figure out where the new capabilities fit best - We go from, this new thing can solve all problems to, here is where this new capability is really good at solving our real problems.

5) Snowflake Will Rediscover Themselves

Although I don’t like thinking in terms of Snowflake vs Databricks….I do have a hard time not comparing them to each other…

As an outsider looking in, Snowflakes vibes are off(as the kids would say).

At their core, they offer a solid data warehouse solution, but their overall strategy, to me, is unclear.

They have heavily relied on partners for a long time, but now, you get the sense that they want to start pushing into other functionalities, while still being friendly with their partners.

Personally, this has led to some of their recent feature add-ons lacking commitment. They could make their dbt integration good, but personally, I find it just fine. I want it to feel less like just tacked on functionality and more like a well integrated part of Snowflake.

I believe Snowflake could implement it in such a way that it could make needing to use dbt Cloud unnecessary. But maybe they don’t want to. They want to straddle both being a partner driven business and an all-in-one data solution.

Databricks, on the other hand, is pushing to solidify its hold on data engineers and analysts. They’ve had a foothold over a good portion of the data engineering identity, and now they are partnering with Alex the Analyst for analytics. This is just marketing.

When it comes to their product, you know Databricks wants to be an all-in-one tool. Sure, they have partners, but they’ve stuck to their core identity; they are all-in-one.

I know I should bring data on this, and I am trying to think what would be a good alternative data source to prove what my gut is saying. But Snowflake gives the vibe that it’s lost it’s way. I really enjoyed reading the book Playing to Win: How Strategy Really Works.

Most of the book discusses the diagram below in the usage of the strategy.

When I overlay that thinking on the Databricks vs Snowflake I can see that Databricks has committed to its choices and where it is playing, where Snowflake hasn’t.

I do think Snowflake will find its way, one way or another, this year.

Leave a comment

Final Thoughts

The data world is still suffering from many of the same challenges it has been for decades. Yes, we’ve added new tools and solutions, but businesses are still trying to find value from their data without getting too distracted by hype.

There are a lot of high-level posts and articles on driving value via data, but I think there is a gap when it comes to speaking on patterns of value that most businesses could find easily.

That’s one of the future series I want to put out after the data pipeline article.

So keep an eye out!

As always, thanks for reading!

Video Of The Week - Common Data Pipeline Patterns You’ll See in the Real World - Types Of Data Pipelines You’ll Build

Articles Worth Reading

There are thousands of new articles posted daily all over the web! I have spent a lot of time sifting through some of these articles as well as TechCrunch and companies tech blog and wanted to share some of my favorites!

What It Actually Takes to Build a Data Pipeline System

When I first started in the data world, it was common that many data teams would build their own data pipeline solutions. There were still dozens of options in terms of off the shelf tools of course, nevertheless, you’d see custom pipelines developed everywhere.

In 2025, I saw less of this.

In fact, in many cases data teams would go straight to picking tools or solutions.

But let’s say you do want to go down this route. You want to build your own data pipeline solution?

How would you do it?

Read More Here

How Uber Scaled Data Replication to Move Petabytes Every Day

Uber prioritizes a reliable data lake, which is distributed across on-premise and cloud environments. This multi-region setup presents challenges for ensuring reliable and timely data access due to limited network bandwidth and the need for seamless data availability, particularly for disaster recovery. Uber uses the Hive Sync service, which uses Apache Hadoop® Distcp (Distributed Copy) for data replication. However, with Uber’s Data Lake exceeding 350 PB, Distcp’s limitations became apparent. This blog explores the optimizations made to Distcp to enhance its performance and meet Uber’s growing data replication and disaster recovery needs across its distributed infrastructure.

Read More Here

End Of Day 209

Thanks for checking out our community. We put out 4-5 Newsletters a month discussing data, tech, and start-ups.

If you enjoyed it, consider liking, sharing and helping this newsletter grow.

Share

Today we’ve got an amazing guest author, !

Olga is an analytics and data science leader focused on building impactful data products and helping businesses turn insights into better business decisions. She brings a strong ability to bridge business and technology and is passionate about mentoring analysts and growing high-performing data teams.

If you’re looking to learn more about everything analytics, then you should check out her newsletter. It’s filled with great content and how-tos.

Now let’s jump into the article!

When I asked Olga to write an article, I wanted it to be focused on skills that data professionals don’t get taught explicitly.

There aren’t a lot of videos out there on how to deliver an impactful analysis to executives.

Even when it comes to running an analysis, many of us likely had to feel around in the dark a bit. I was just speaking to another data science leader who said they had to have an executive essentially take them aside and let them know their analysis weren’t great.

There are so many of these skills that analysts and engineers alike have to pick up on the job and no one tends to tell you what is good or bad.

So let’s talk about some of those skills you need to start working on!

1. How To Develop Analytical Intuition

Many companies will ask candidates questions that might seem out of the blue. Like, how many dentists are in the world?

What they are trying to gauge is your analytical intuition.

Essentially, given problem with limited information, can you come up with a reasonable framework or approach to answer the question or at least know what information you’d need to look for in the future.

Here are some tips if you’re working to improve your analytical intuition.

Ability to set reasonable ranges and break down sampling:

Example: How many windows are in NY city? How many teachers are there in the world?

How to: Start with an educated guess based on something related to the question — a proxy value that you do have some intuition about. Then, work your way toward a ballpark estimate using averages and scaling logic.

Critical thinking: What goes up must come down

If a value goes up, it should come down by a similar degree at some point, and vice versa. That’s why we use continuous distributions and probabilities - to account for natural variance, not just isolated changes.

If you’re new to a dataset or project, the first thing you should do is understand the degree of natural traffic fluctuations and baseline variance. This helps you separate expected changes from those driven by external factors and understand how much deviation is “normal.”

Math and fractions: Part of a whole

Every metric is a fraction. It’s just one piece of a broader ecosystem made up of other interconnected parts. Let’s say you confirm that the payment success rate is 25%. That means 25% of users successfully complete a transaction. But it also means that 75% do not. The more related metrics you identify, the easier it is to cross-check them.

This same logic applies to funnels and conversions. Every metric is a fraction of a whole. If one thing is rising, something else (a) should be falling or (b) should be rising as well. If you don’t see (a) or (b), question everything. Once you understand the relationships between metrics, figuring out what is declining and why becomes much easier.

Develop a habit of checks: random users, session flow, against diff tools

For every sample or report, pull 5-10 random users from the dataset and manually check their attributes - paid price, invoice details, subscription plan, country, number of transactions, etc.

Build the habit of manual spot checks and cross-checks in every report or dashboard. Don’t trust tooling or automation.

2. How To Do a Root Cause Analysis

We apply root cause analysis when a metric shows unexpected movement. For example:

• Average DAU starts gradually declining M/M, but MAU stays flat.

• Net new transactions increase 5% W/W, but total revenue doesn’t change.

• Customer churn for annual subscriptions doubles.

• The trial-to-paid rate drops by 10%.

The keyword is unexpected. Analysts own the concept of a baseline - an estimated (modeled or forecasted) value with applied seasonal, M/M, and Y/Y adjustments.

For example, you may notice that total transactions sharply decline in September after August. This decline may be expected if the same pattern appears each year during this period. Or it may point to a bug. Or it may be a mix of both. To break it down, you need to know your baseline -how many transactions you typically expect at this time of the month and year, and how much that number has changed.

First things first: ensure the data you see is correct.

Find at least two or more other data sources showing a similar decrease to confirm it’s true.

Possible issues: broken ETL, holiday schedules that break during long weekends, etc.

If you’re confident the data is accurate, and this is an actual decline, proceed with modeling different hypotheses on what the issue may be.

Analysis: Generate multiple hypotheses that you have to confirm or reject.

1. Product hypothesis: the drop is related to a product bug or a specific product launch. There is usually a sharp drop if it’s a bug. The drop can also be gradual for releases because teams often do slow rollouts with 1% traffic release, then 20% → 50% → 100%. It can also be for a segment of users (e.g., new users only or paid).

2. Market or competition hypothesis: it can be a new tool that takes the market share. Or slowed down spending, or a shift in user acquisition strategy. You will notice it by a gradual decline, not necessarily tied to a specific campaign or promotion.

3. User hypothesis: This is typically a gradual, inconsistent decline. Given that (a) the proportion of different personas tends to change, and (b) they are not necessarily tied to seasonality (rather to marketing campaigns that brought them in), it can be challenging to capture the beginning of a decline. You would need to check the drop against different user cohorts - registered but not active, active but lapsing, power users, premium or free, business or individuals, or whatever personas you work with.

4. External factors: pandemic, war, or big social movement (#MeToo, BLM, abortion law, etc.). These may cause a sharp decline (or an increase) in usage across many top user actions, features, and platforms.

Once you’ve proven the hypothesis based on the data, whether it’s a product bug, marketing issue, or user behavior, escalate it to the appropriate team to gather more context. Ideally, you’re already a few steps ahead, with hypotheses and analyses prepared on which metric is declining and by how much.

3. How To Develop A KPI And Connect It To Action

KPIs can be financial, customer-focused, and process-focused.

There are many types of metrics:

• Top-level metrics - a measure of strategic direction and performance. Each top-level metric is tailored and adjusted to the specific business model, environment, and strategy. This is when context and nuances really matter. Top-level metrics and KPIs are usually reported monthly and quarterly.

• North Star Metric - represents one company goal. From Mixpanel North Star Metric: “To qualify as a “North Star,” a metric must do three things: lead to revenue, reflect customer value, and measure progress” It can be DAU, or LTV, or MRR, or other measurements, and its main purpose is to align teams around one main goal.

• Secondary metrics - more granular health indicators and product targets. They measure how successful the product and process are. Secondary metrics are sensitive to any product changes. Therefore, you would measure A/B tests, feature adoptions, or bug impacts against secondary metrics. Due to their sensitivity, they are also usually monitored and reported weekly.

• Vanity metrics - impressive but not useful or actionable metrics that don’t lead to growth or revenue and aren’t relevant to anything you can do to improve them. They are often too simple and ignore the context. Examples: the number of social media followers or total registered users. It’s like only working out your arms when you go to the gym and ignoring your core. More about Vanity metrics.

• OMTM - One Metric That Matters. This is different from the North Star Metric and meant to be a temporary goal unifying all the teams at the company towards one issue. An example: when your software got hacked and all user accounts got deleted, you would set OMTM as a number of reinstated accounts. For a less dramatic example, when you begin a migration, your OMTM can be the number of successfully migrated accounts. Or when your churn significantly overweights new accounts and renewals, then you have to pause everything and focus on retention.

The right metric should be:

• Relevant - represent the result you want to see. If you make a change to a user flow, you should measure user steps and following actions, not net new revenue.

• Measurable - do you even have the data to get the metric? Do you trust the source?

• Specific - detailed to illustrate the right product movement. User retention is not the right metric to measure the A/B test. But the frequency and/or type of actions are.

• Prioritized - what stands out from other metrics as the highest priority. How to differentiate nice-to-have metrics from must-have metrics reporting.

• Balanced - meant to measure positive and negative outcomes. If you notice a traffic increase for using one feature, most likely there is a decrease somewhere else.

Metrics types

You probably know it already, but there 4 main categories of metrics that are meant to capture different purposes:

1. Sums and counts - Daily Active Users, the sum of sales, unique number of unsubscribers, etc.

2. Distribution (mean, median, mode, percentiles) - average memory used, % of MAU, a median session length, or others.

3. Probability and rates - if you change a screen layout, you have to measure click-through rate or click-through probability.

4. Ratios - monthly/annual subscription ratio, male/female usage ration, or etc.

Here are a few examples of common metrics across different domains:

Growth & Marketing: Unique Visitors, First Visits, Returning Visitors, Bounce Rate, Installs, Signups, Customer Acquisition Cost (CAC), Click-Through Rate (CTR), Cost Per Impression (CPI), Cost Per Action (CPA), Time to Value, Visitor-to-Signup Rate, Signup-to-Payment Rate, Product or Feature Adoption Rate, Virality, Network Effect Score, Return on Advertising Spend (ROAS), Number of Qualified Leads, Lead Conversion Rate, Average Lead Score, Cost Per Lead (CPL), Unsubscribes.

Revenue: Monthly Recurring Revenue (MRR), Annual Recurring Revenue (ARR), Net Revenue, Net Revenue Retention, Paid Customers, Activated Trials, Free-to-Paid Conversions, Paid-to-Free Downgrades, Revenue Churn, Customer Churn, Monthly/Weekly Customers Completing Their First Order, Daily/Monthly Total Purchase Value, Lifetime Value (LTV), Average Revenue Per Account (ARPA), Upsell-to-Payment Rate, Expansion Revenue, Return on Investment (ROI), and others.

Engagement: MAU, WAU, DAU, Adjacent Users, Day 0, Day 1+, Day 7+, and Day 28 Retention, 1-Year or 2-Year Retention %, Number of Returning Users, Daily/Hourly Number of Actions, Total Watch Time, Total Time Spent, Frequency of Visits, Pages Per Session, Scroll Depth, Average Session Duration, Exit Rate, Product Abandonment Rate, and others.

Customer Success: Customer Satisfaction Score (CSAT), Net Promoter Score (NPS), Customer Health Score, Ticket Resolution Rate, Average Resolution Time, Average Reply Time, Customer Effort Score (CES), First Response Time, Daily/Monthly Ticket Requests.

Platform / Engineering: Product Support Cost, R&D Engineering Cost, Outsourcing Rate, Cost Performance Index (CPI), Schedule Performance Index (SPI), Uptime, Average Downtime per Month/Year, Machine Downtime Rate, % Planned Maintenance, Number of Releases, Running Cost, Number of Bugs, Number of Pull Requests, Capacity Utilization, Memory Usage, Requests Per Minute (RPM), Errors Per Minute, and others.

If you’re like to read more on KPIs:

• Intro to Product Metrics

• How To Pick The Right Metric

4. KPIs Done Wrong: Fixing Common Reporting Mistakes

1. Don’t overthink KPIs. If you’re unsure how to measure an initiative, stick to simple metrics like unique views, CTA clicks, and the % of users with a CTA click. Not everything needs to be tied to LTV or MRR.

2. Use an effective proxy metric that is both sensitive and independent. If your proxy requires complex calculations, it’s not a good proxy.

3. Don’t stress about benchmarks - focus on your Signup-to-Paid MoM growth.

4. Avoid bringing metrics definitions from your previous job into your current project. Every product is unique, with different user lifecycles. Some apps have monthly and annual subscriptions, while others have 18 payment plans. Churn calculations will vary. Develop KPIs that fit this specific business and product.

Final Thoughts

There are a lot of skills you’ll start to take for granted as you grow as a data analyst or engineer. But they all came from somewhere.

I hope this newsletter helps you put words to concepts or can be an article you share with someone just joining the data world.

As always, thanks for reading!

Articles Worth Reading

There are thousands of new articles posted daily all over the web! I have spent a lot of time sifting through some of these articles as well as TechCrunch and companies tech blog and wanted to share some of my favorites!

Apache Hudi™ at Uber: Engineering for Trillion-Record-Scale Data Lake Operations

Uber operates one of the most diverse and demanding data ecosystems in the world. Every trip taken, order delivered, ad served, or real-time arrival time recalculated generates an unending stream of data. These data points come from hundreds of microservices, thousands of cities, and millions of riders, each with its own velocity, shape, and business-criticality. At the heart of this ecosystem lies Uber’s data lake: a multi-hundred-petabyte repository that fuels operational decisions, machine learning models, experimentation platforms, and real-time business intelligence.

Read More Here

What It Actually Takes to Build a Data Pipeline System

When I first started in the data world, it was common that many data teams would build their own data pipeline solutions. There were still dozens of options in terms of off the shelf tools of course, nevertheless, you’d see custom pipelines developed everywhere.

In 2025, I saw less of this.

In fact, in many cases data teams would go straight to picking tools or solutions.

But let’s say you do want to go down this route. You want to build your own data pipeline solution?

How would you do it?

Read More Here

End Of Day 208

Thanks for checking out our community. We put out 4-5 Newsletters a month discussing data, tech, and start-ups.

If you enjoyed it, consider liking, sharing and helping this newsletter grow.

Share

Today I am continuing my series on data pipelines. In the prior article we discussed the types of data pipelines that exist. Today, we’ll be discussing the components you’ll need if you plan to build your own data pipeline from scratch.

But before we jump in, I wanted to share a bit about Estuary, a platform I’ve used to help make clients’ data workflows easier and am an adviser for. Estuary helps teams easily move data in real-time or on a schedule, from databases and SaaS apps to data lakes and warehouses, empowering data leaders to focus on strategy and impact rather than getting bogged down by infrastructure challenges. If you want to simplify your data workflows, check them out today.

Now let’s jump into the article!

When I first started in the data world, it was common that many data teams would build their own data pipeline solutions. There were still dozens of options in terms of off the shelf tools of course, nevertheless, you’d see custom pipelines developed everywhere.

In 2025, I saw less of this.

In fact, in many cases data teams would go straight to picking tools or solutions.

But let’s say you do want to go down this route. You want to build your own data pipeline solution?

How would you do it?

What Components You’ll Need

Below I’ll outline the components most every data pipeline system I’ve worked with requires/has had.

Secrets And Connection Management

I am going to start the list of components off with secrets and connection management.

Because this is how you’ll likely set up source and destinations, without sources and destinations, you really have no reason to build your pipeline.

You just have orphaned SQL logic doing nothing, and Python pushing data nowhere.

It’s also crucial in how easy you make it to manage the rest of your system.

Do you want your data team members to have to write a custom connection script every time?

If a source or password changes, are you making it easy to update the information in a single place or multiple places?

Do you make it easy to store securely without exposing it to the repo?

Small details here matter and add up over time. If I have to make a separate connection reference every time I need a new table from the same database, that’ll be terrible.

And for those out there who assume they’ll only ever need to connect to a few sources, you’d better hope you’re right.

Logging And Monitoring

When you build your first pipeline system, maybe you put in a few print statements to track where your pipeline has succeeded and failed. As you start building a more generic system, logging needs to be included.

You need to be able to trace back and figure out if there was an issue with an external library, with a specific module inside your data pipeline system, or an actual problem with a pipeline you’ve written.

Think “we can’t find this library” vs “we can’t find this table”.

You also want to know on which run this occurred. Was it data from a specific date, or if you think in terms of Airflow, one of those little red boxes?

Without logging, debugging is impossible, and as more code gets generated by AI, we are going to need even more specific error messages and traceability in order to figure out what we need to fix.

Dependency Awareness(Graphs)

Not everything has to be DAGs, but you’re going to need some sort of dependency awareness. I recall building a very naive version of this at one of my first jobs, where I simply created a table that kept track of which jobs had run, on what date, and the numeric step in the process.

This quickly falls apart if your pipeline needs to change frequently or you need to build anything with a smidge of complexity. Then you start needing to look at solutions like Airflow and dbt, and how they handle referencing dependencies.

For example:

• extract_orders.set_downstream(transform_orders)

• SELECT * FROM {{ ref(’stg_orders’) }}

Some how, you do need to tell pipelines what the prior task or tasks they need to wait on are so they can check the status.

Execution Engine Routers

I’ve seen multiple companies now spend much of their data teams’ budget and time simply on migrating data from Databricks to Snowflake.

Why?

Because they use Databricks to run the expensive, heavy early data processing and then use Snowflake as a service layer.

Data teams want to be able to pick the compute they need, and so this is somewhat of a newer concept for data pipeline solutions. We now have multiple compute engines that people want to use when processing data. Think DuckDB, Presto, and maybe just a local instance of Spark.

Some engines are cheaper or faster, and still others handle larger data sets better. I foresee solutions in the future, routing more and more of this traffic to optimize for what your team is looking for in a specific pipeline. We actually had this at Facebook(although we had to tell it which engine to use)

In the same way, it’s worth considering if you ever plan to build your own pipeline solution. I wouldn’t build it right away. But if you’re looking to further optimize your own internal solution or if you’re thinking bigger and building a dbt competitor(which several people have reached out to me saying they are), then I’d consider adding in routing.

Scheduler

Not all pipeline tools have a method to schedule the jobs you’ve built. In fact, I’d guess many don’t. If you’ve used SSIS, you’ve likely had to use SQL Server Agent. I’ve seen people use Jenkins, and of course, Cron and Windows task scheduler.

So even if you don’t have a scheduler in your pipeline system, it’ll have to be somewhere. Something has to tell your pipeline when to run.

If you look at Airflow’s built-in scheduler, it works differently from tools like cron or SQL Server Agent.

Instead of directly triggering jobs at a specific time, the scheduler continuously evaluates which DAG runs should exist based on logical time and dependencies. Once a DAG run is created, the scheduler determines which tasks are eligible to run and hands them off to executors and workers to do the actual work. This design separates when work should happen from how it runs, which is what enables backfills, retries, and complex dependency management.

But likely you’re starting off via Cron.

Pipelines

I’ve spoken a lot about the components around the actual pipelines. But, well, you need data pipelines.

You need a way to define them.

You can look at tools like Glue, dbt, SSIS, and Airflow for possible examples. dbt is only a part of a pipeline, but it still does much of what I’ve listed above. Airflow, on the other hand, could be viewed as a much broader tool(a workflow orchestration solution).

In all the cases, you have a way of defining your pipeline(or workflow).

In dbt you might view this as several models(SQL) leading to a final table, paired with a solution that can ingest your data.

These days you can often find pipeline solutions that mix code and drag and drop together. Where you might use dbt and some other tools together.

Data Quality Checks

When I started building my first data pipelines, I had to write a lot of tests by hand, well, type them. We didn’t set easy-to-configure checks that you could simply reference like:

rowcount_bounds = SQLValueCheckOperator(

task_id=”dq_rowcount_bounds”,

conn_id=”warehouse”,

sql=”SELECT COUNT(*) FROM fct_orders WHERE ds=’{{ ds }}’”,

pass_value=120_000,

tolerance=0.2,

)

Having an easy way to integrate data quality checks as part of your pipelines is crucial.

Sure, you can find other tools that offer some broader set of checks, but I think having some that are already easy to integrate into your pipeline ensures your team will integrate data quality into your pipelines. Because if it’s not easy to integrate, people won’t do it.

UI

Unlike many of the other components, it could be argued that a UI is optional. You could just run your pipeline via a CLI, where you also check statuses.

But if you are going through all the trouble of building this data pipeline tool internally, it likely means you’re supporting a rather large data team.

If that is the case, then a UI becomes more necessary as there are other teams that need to interact with said UI. They need to filter jobs to see what jobs are running, what jobs are hung, and which ones have failed.

They need a place where it’s easy to re-run jobs that are stuck, track logs and alerts, and so on.

Now suddenly, you’re building an entirely new application.

Processes And Operational Concerns

I don’t consider these next aspects components, but there are processes and operational concerns you need to consider when building data pipelines. Here are a few.

Idempotency & Backfilling

When you first start building pipelines, it’s easy to build them only to function one way. That is to say. You build them to load data in, transform data, and be right once.

If you were to rerun it again(which we often call this backfilling).

Guess what, now you have duplicate or incorrect data. Often, you have to manually go back and delete the data prior to re-inserting it.

So, as you build your data pipeline system, you need to make it easy for data engineers to rerun your pipelines safely. Sometimes this is done because you’re storing data via date partitions and just insert overwrite, other times you ensure that the pipelines themselves delete data automatically or check for duplicate data either via IDs or dates.

The more intuitive you can make this, the better. That way, future engineers can rerun without having to check 1000 different places to ensure your pipelines ran as expected.

Ownership

Another aspect I believe pipelines need is a way to track ownership. This really should just be a configuration inside the pipeline itself. It’s one reason I like the fact that Airflow lets you define this(which Facebook also had). This makes it easier to track down who may have either built or currently owns the logic.

Without it, pipelines can quickly become orphaned and when you need to find out who built it, you’ve got to go to six different places to see if you can find the last person that updated the code.

Alerting and on-call routing

Along with ownership comes alerting and on-call routing. As a company grows and as teams become decentralized, data pipeline failures won’t just go to a single team.

Meaning you need a way to handle who or which team gets alerted when pipelines fail. Also, as data pipelines become more crucial to the business, on-call becomes more important. There are some pipelines that can likely fail and have someone deal with it in the morning.

On the other hand, you have pipelines that will fail that require attention immediately. That’s where on-call routing comes in. You want to not only alert your data team, you want to alert the right member who is likely…sadly…on-call.

Environment Isolation and Promotion

As you’re building out your pipelines, you’re going to want to make it easy to point your pipelines at either development, testing, or production(or however you’ve broken down your environments).

Actually, at Facebook, test pipelines created tables in the same environment as production ones. The difference was that when you ran pipelines as a test, they would automatically set the prefix of the table to “test_” whether you added it or not.

Now you could override this and use something like say “rogo_test_” which I did from time to time. But overall, I’d say most people ran in this setup quite well.

Whatever set-up you pick, my recommendation here is that you make it part of the natural developer flow. If it requires too much switching over from test to dev to production. Someone is dropping a production table(of course, this is less of an issue with many cloud platforms)

There are still other components but these are the main ones I’d consider if I were building a solution from the ground up(And I know I’d find a half dozen more as I started building)

Final Thoughts

There is a wide spectrum of solutions you can use to get data from point A to point B to allow a larger group of people to run analytics. You could build a simple set of Lambda functions to extract and load data, use out-of-the-box ELT solutions, or orchestration tools.

Many data teams likely don’t need to build their own entire data pipeline system from scratch. There are always exceptions, of course, and hey, maybe you are just looking to build out the next Airflow or dbt.

If you are, think through the full scope, how much of the data pipeline and orchestration spectrum do you want to cover? Are there ways you could improve a developer’s flow that haven’t been integrated prior, such as with AI or maybe with routing to other compute engines?

I also wish you the best of luck, as it is quite a large problem space to cover!

Thanks as always for reading!

Articles Worth Reading

There are thousands of new articles posted daily all over the web! I have spent a lot of time sifting through some of these articles as well as TechCrunch and companies tech blog and wanted to share some of my favorites!

Behind the Scenes of SQL: Understanding SQL Query Execution

Here is something school probably didn’t teach you about SQL

Or at the very least you likely forgot.

When you write a query, hit submit, and then run the query or that little triangle in DBBeaver…

…what exactly happens?

Sure, you likely understand that data is pulled from multiple tables, data is filtered, and aggregations occur.

But behind the scenes, what is going on?

How does SQL go from English into the lingua franca of data?

In this article we will answer that question.

Read More Here

Back To The Basics: What Is Columnar Storage

Data engineers often discuss columnar storage, especially when discussing data warehouses or file formats like Parquet.

But why does it matter?

Why is columnar storage such a big deal for analytics? What makes it so well-suited for analytical use cases compared to other formats?

Maybe you already have a rough idea.

This article will help further explain columnar storage, why it’s used, and how it compares to row-based storage. It will also look at the most common formats and real-world use cases.

Let’s dive in.

Read More Here

End Of Day 207

Thanks for checking out our community. We put out 4-5 Newsletters a month discussing data, tech, and start-ups.

If you enjoyed it, consider liking, sharing and helping this newsletter grow.

Share

This is the first newsletter for 2026!

One of my goals in 2026 is to put together series. So this is the first of a longer series focused on data pipelines. I wanted to start out by discussing the types of data pipelines I’ve seen in terms of how they are used as data pipelines can be used for more that one specific use case.

Before we jump in to talking about backfills, I wanted to share a bit about Estuary, a platform I’ve used to help make clients’ data workflows easier and am an adviser for. Estuary helps teams easily move data in real-time or on a schedule, from databases and SaaS apps to data lakes and warehouses, empowering data leaders to focus on strategy and impact rather than getting bogged down by infrastructure challenges. If you want to simplify your data workflows, check them out today.

Now let’s jump into the article!

Whether you’re working at a large enterprise or a small business, there has likely been some need to take data out of the various source systems, process it, and then use it for either operational or analytical purposes.

Add in a few lines of code or a low-code solution, and the term data pipeline might start getting thrown around.

This might make some data engineers angry, but if you think about it, someone extracting data from a data source into Excel, adding in VLOOKUPs, some data cleansing via formulas and IFELSE() statements is essentially building a data pipeline….

Ok, it’s not the exact same thing, but when you stop and think about it, it can functionally solve a similar problem(although often in a more limited and specific way)

My point is that there are a lot of different ways and reasons people build data pipelines.

So, to kick off 2026, I wanted to discuss some of the key reasons data pipelines exist and the types of pipelines you will run into.

Source Standardization Pipelines

Some of the first pipelines I helped build and manage were focused on taking data sets from dozens of companies and standardizing them to a single core data model. In particular, this involved getting data via SFTP in different formats, including comma-delimited, pipe-delimited, XML, and even positional files, where you had to have a separate file that would define which columns contained which rows.

This might be unfamiliar to data engineers who are accustomed to build data pipelines to answer questions around SaaS products such as retention and churn.

But this is a problem I’ve run across now many times across many different industries from health care to retail and real estate to name a few.

In many cases this wasn’t even purely for analytics. The centralization and standardization of the various data sets allowed the companies to provide operational benefits or other services. For example, maybe you’re trying to create a marketplace and need to centralize dozens of different inventory sources.

The challenge when building these data pipelines is usually that amount of effort required to onboard and actually create scripts that can manage all the variations of how different data will come in. This is referred to as mapping.

You’ll need to:

1. Standardize values such as gender which can often come in as a number, single letter or the written out word

2. Standardize on categories, I’ve seen this a lot in retail where products might be in the same category, but one might use an abbreviation or a different word that means a similar thing

3. Fix date and format inconsistencies, such as different time zones, different date formats, or missing values entirely

And of course more such as how each data set might be appended. You can mitigate some of this by asking your external partners to send data in a way that is standardized, but it’s difficult to fix every issue.

Once you do have a standardized data set, you can build multiple products off this data. Whether it be a marketplace or an industry level report and because it’s all standardized it’s easy to apply new products and features for all your customers.

The one final point I will add is that this is not just limited to SFTP data sets, I’ve worked with companies that pull in data from APIs as well.

Read more

Maybe it never started that way.

Maybe when they first came out, they only ever planned to be a managed Spark solution. But I have a hard time believing that, mostly because I believe their leadership has the vision and capabilities to see far beyond that.

Databricks has always been pretty upfront that they want to be the end-to-end data stack. But they’ve been approaching it piece by piece.

Or should I say role by role?

Obviously, at first, their focus was on the data scientist and ML engineer.

But in 2020, they wanted to shift the narrative. They were more than just managed Spark, they were a data platform that could replace your others. So they championed the idea of the Data Lakehouse. You could view this as a capturing of the data engineering market.

But this didn’t happen overnight.

If you look at some of the posts and content shown below, Databricks tends to push an idea or concept very hard when they really want to capture that market. Shown below when they pushed the concept of the Data Lakehouse and now their data visualization and analytical workflows.

I’ve already written about the Data Lakehouse and how it was heavily pushed by Databricks in 2020, so I don’t want to dive too deeply here. I believe there are far more interesting recent happenings that stick out.

This past month, I believe Databricks started to make its final push into gaining the mental market share for analysts.

How?

Databricks just partnered with the biggest analytics creator.

Alex The Analyst.

There are several reasons I believe this is significant.

1. Although many companies partner with data creators all the time, I think, if we look at Alex the Analyst’s audience, we’ll see that Databricks wants to put the final nail in the coffin of the data analytics workflow. They want to show end-users that hey, if you’re an analyst, Databricks is for you too! We aren’t just for data engineers; we can help you drive value, and we aren’t too complex.

2. Databricks did something similar when they wanted to win over the Data Engineering space. They partnered with Bill Inmon to put out a seminal piece on the Data Lakehouse. This was about a year after Databricks wrote their original piece, and it wasn’t until this piece came out that they finally got real traction.

3. Databricks has been pushing its data visualization solution for a long time now. But now, they want to get even better traction by getting the face of learning analytics out there.

4. I wouldn’t be surprised if one reason they are doing this is that they are likely continuing to get pushback from customers who are more of the analyst type, who believe Databricks isn’t for them. In fact, I have worked on several projects over the past year alone where that was the sole reason for a Databricks and a Snowflake environment.

In these cases:

• Snowflake was for the analysts.

• Databricks was for the engineers.

I think the other point that stands out is how heavily the Databricks visualization tool was shown in Alex’s content. Now, part of this could have been because that’s what Alex felt his audience would appreciate.

But here is another anecdote I’d like to share.

I’ve spoken to several data leaders who have told me that Databricks account executives tried to push them very hard on their visualization tool. In some cases, to replace their Tableau or Power BI instance.

I think account executive interactions tell you a lot about a company’s strategy. They are the front line; what they are doing is literally a reflection of where the company is trying to go. If the company wants to push a new feature, a new partnership, or a new product, you push it through the account executives.

What Does This Mean For The Data World?

Ok great Ben, why does anyone of this even matter?

Here are a few thoughts on where this is all going and why it’s important.

Snowflake Vs Databricks Is A Red Herring

The real vs. conversation isn’t Snowflake vs Databricks, it’s really Databricks vs Microsoft, AWS, and even Salesforce.

Let me explain.

These are more generic clouds and SaaS, but they also offer either components or the entire data stack.

More importantly, they are often solutions you pick prior to thinking about your data tooling. You likely use Azure to set up your application or Salesforce as your CRM. So, when it comes time for data analytics, you already have Azure, so just use Power BI; it’s already part of your contract anyway (I’ve had multiple data leaders give me that line of thinking when it comes to picking BI).

Why not use Tableau? You’re already using Salesforce. Will it probably make negotiations better, right?

Of course, there is still some Snowflake vs Databricks. But for those 8 and 9-figure deals, you’re fighting much larger companies. Companies that could buy you out.

But this leads to my next point.

Databricks Is Building The Next SAP

I referenced this idea at the bottom of a post on the Data Leaders Playbook that Snowflake and Databricks are building SAP backwards. Instead of going from business applications and eventually building out solutions like SAP HANA.

They are going from data analytics to the business.

With the recent purchase of Neon, Databricks can now enter the sales conversation earlier, which I think is arguably the real point. Yes, yes, all the tech people are pushing back their glasses and about to say something pedantic.

If you want to win the CIO over, and not just the data team, you have to enter the conversation earlier. Meaning the application layer. I am sure some Palantir fans in the back are just waiting to start writing a comment(This has been Palantir’s goal the whole time!).

Let’s put that aside for now and just think about the company these businesses are selling into. Most companies, especially digital ones, have a few key databases. The ones that actually represent their main application or service. It’s where most of their data lives. If Databricks can get you to use their Postgres instance, then they already have you in the funnel to get you to use their data Lakehouse, then they already have you in the funnel to use their BI, their data pipelines, etc.

So I expect they will be pushing this space hard in the next year or so. I’ve actually already seen this hinted at via some of their distribution and creators they enjoy working with.

If You Want To Win Mental Market Share You Have To Be Relentless

Databricks was pushing the Data Lakehouse concept for over a year until it really started to gain traction. I imagine it required millions in terms of employee time, partnering with consultants, etc.

You can’t just create a term or want to gain entrance into a space without having a plan on how to win.

I say this because I’ve spoken with so many marketing teams and leaders who have wanted me to do “un-boxings” of their product as a one-off.

That’s a tactic, not a strategy(and not a very well thought out one at that).

One video doesn’t move the needle. Think about how many posts, videos, talks, consulting partners, and so on, Databricks has been pushing to talk about their AI/BI Genie and Data Visualization.

Just paying for distribution on a one-off piece of content won’t work.

Even now, with the recent partnership with Alex. This doesn’t feel like a one-off. Databricks keeps promoting the content. On their LinkedIn page, in their product.

They’ve got the employees posting about it. They are trying to make Fetch happen, and they know this isn’t going to be easy.

They want analysts to come to Databricks, and when they do, they want them to see a familiar face.

If done well, it can pay off. I doubt you can fully attribute Bill Inmon to shifting 100% of the conversation to the Data Lakehouses, but if he even had a 5-10% impact, that’s a multi-million dollar impact.

Databricks wants to do that again with the analyst.

Final Thoughts

Something tells me that Databricks will be pushing Alex’s videos hard for the next few months.

It’ll complete the trio of data roles.

In turn, this allows Databricks to start to focus on the application layer.

To start having conversations earlier with business, not just about data strategy, but about IT and business strategy.

As always, thanks for reading.

Video Of The Week - 5 Things in Data Engineering That Still Hold True After 10 Years

Articles Worth Reading

There are thousands of new articles posted daily all over the web! I have spent a lot of time sifting through some of these articles as well as TechCrunch and companies tech blog and wanted to share some of my favorites!

Is It Time to Say Goodbye to Data Engineers?

Ever since tools like SSIS came onto the scene, vendors and business leaders have been on a mission to remove what they see as the biggest roadblock to data-driven decision-making: data engineers.

Or their counterparts—DBAs, ETL Developers, and Data Architects.

Sure, not everyone says it so explicitly, but you can see it in vendor marketing and in the decisions made by the business.

I remember talking to a veteran data expert who’s been in the field for three decades. They told me that when SSIS first launched, people were genuinely afraid for their jobs. The idea that you could just drag-and-drop tasks that once required code was nerve-racking. But if you’ve used SSIS, well, you know the truth.

To some extent, I get why the idea is appealing. When a leader requests a report, a software engineer wants to modify an application table, or a data scientist wants to explore a new dataset, who’s the one slowing down the project?

The data engineers.

Read More Here

Refresher on Experimentation

By

As has become a tradition, I end the year with a refresher: one single, consolidated guide that brings together the most important definitions, reporting best practices, example dashboards, and key principles - all in one place.

Last year, I published these three:

1. Refresher on Retention

2. Refresher on Statistics

3. Refresher on SQL for Data Analysis

This week, I’m sharing a refresher on A/B Testing. I believe I published one before, and I also have a dedicated experimentation section in my newsletter. This time, I expanded it with more resources to help you properly get started with A/B testing. It covers tools, core concepts, free classes, and more.

Read More Here

End Of Day 205

Thanks for checking out our community. We put out 4-5 Newsletters a month discussing data, tech, and start-ups.

If you enjoyed it, consider liking, sharing and helping this newsletter grow.

Share

Today we have another amazing guest author.

Mehdi Ouazza —better known as —is a data engineer and developer advocate at MotherDuck with over a decade of experience building data systems at companies like Klarna, BackMarket, and Trade Republic. He combines deep engineering expertise with a playful, approachable style, creating blogs, videos, and social content that make complex data and software topics easier to grasp. When he’s not shipping pipelines or crafting educational content, he’s usually experimenting with creative ideas that bring a fresh, off-beat energy to the developer community.

He’ll be sharing how he runs system design interviews for data engineers, but I won’t get too ahead of myself.

So let’s jump into the article!

After hundreds of technical interviews for data engineering positions, I’ve developed what I call the “exploration” approach to system design interviews. Think of it like exploring a new city: you start with a bird’s-eye view of the whole map, then zoom into specific neighborhoods that look interesting, getting to know some streets really well while keeping track of how they connect to the bigger picture. It’s become my secret weapon for truly understanding a candidate’s technical knowledge while keeping the conversation engaging and productive.

This type of interview matters even more in our current AI-assisted world. Coding matters less than ever before. We can generate vast amounts of code with tools like Copilot or ChatGPT. But understanding how components work, their foundations, and their trade-offs? That’s irreplaceable.

If you’re interviewing with me, consider this your cheat code.

Setting the stage

Be human

You would be surprised how many interviewers miss the obvious: presenting yourself as a human being. I’ve sat through countless interviews where the interviewer jumps straight into “I’m the head of engineering, let’s talk about Kafka.” No. Just no.

When I start an interview, I actually introduce myself: where I’m based, how long I’ve been with the company, maybe share that I once debugged a production issue on my phone while biking (true story). This isn’t fluff, it’s creating an environment where candidates can relax and show their best selves. Interviews are stressful enough without feeling like you’re talking to a robot, especially over a video call where human connection is already harder.

I also explicitly tell candidates upfront: “We’re going to start really broad with the whole system, then zoom into specific areas for technical depth. It’s completely okay if you don’t know everything: nobody does. I’m more interested in how you think through problems than memorized answers.”

This transparency matters. Without it, candidates might think “Wow, this is easy” when I ask the initial broad question, only to panic when we drill into the internals of Parquet files.

That’s not fair to them, and it doesn’t help me assess their true capabilities.

Live white boarding collaboration

I use Excalidraw for live collaborative drawing: it’s free and requires no setup/login. Candidates can visualize their architectures while we discuss them. This isn’t just about pretty diagrams; it’s about thinking visually and communicating complex systems clearly.

I actively support candidates during the interview. If they get stuck, I’ll introduce a source system or clarify a requirement. This isn’t a gotcha test : it’s a collaborative design exercise, much closer to real work.

After all, how often do we get perfectly clear requirements from stakeholders? 😉

Wide & zoom through 3 core scenarios

I need someone who understands how components connect, why certain architectural decisions matter, and can demonstrate real depth in at least some areas, depending on the requirements of the role.

The exploration metaphor works because we’re mapping out their knowledge landscape together.

I usually cover 3 main system design scenarios, each highlighting different aspects of data engineering. In a 45-60 minute interview, there’s rarely time for all three. If a candidate shows strong depth, I move on quickly; if they struggle, I spend more time there. The goal is to map both strengths and weak spots.

1. Simple ETL data stack

I start with something deceptively simple: “You have a small e-commerce website, and you’re getting data that you need to analyze. Design your first data infrastructure. How would you build the data stack?”

The question is intentionally vague. Good engineers don’t just dump technology names; they ask questions, make assumptions, and tailor solutions to specific needs. This is where the real engineering happens!

Some might argue: “Why do I even need a data stack? I can just query my current database!” And that’s a perfectly pragmatic answer.

Then I’d evolve the scenario: “Now imagine you have a table larger than 100GB and need to run analytics queries. Would you still rely on the same setup? If not, how would you redesign the stack, and why?”

When candidates sketch their initial architecture (ingestion → processing → storage → serving), I probe further based on the technologies they mention. Here are some common examples:

File format deep dive

• “You mentioned storing data in object storage. What format would you use?”

• “Why Parquet over CSV?”

• “If Parquet is so fantastic, what are its limitations?”

• “I see Iceberg and Delta Lake mentioned everywhere. What problems do they solve that Parquet alone doesn’t?”

This progression reveals whether someone just memorizes buzzwords or truly understands the trade-offs.

Here’s how a typical deep dive might unfold:

The data warehouse vs Lakehouse debate

• “Should compute run inside or outside your data warehouse?”

• “What are the implications of each approach?”

• “If running inside, what should you watch out for? Indexing? Data volume?”

These questions separate those who’ve actually built and operated systems from those who’ve only read about them.

2. Streaming pipelines

Next, I change the requirements: “Now I need 1-5 minute latency between source data and where the data is being served. Redesign your solution.”

This shift from batch to streaming reveals a different dimension of understanding:

The Pub/Sub Deep Dive

• “You mentioned needing a pub/sub system. Which ones do you know?”

• “How does Kafka actually work? What are brokers, partitions, and consumer groups?”

• “What’s the difference between at-least-once and exactly-once semantics?”

• “How do you handle late-arriving data in streaming?”

Again, I’m looking for practical knowledge here. Anyone can say “use Kafka,” but understanding partition strategies, offset management, and the challenges of maintaining streaming infrastructure shows real experience.

3. Customer-facing analytics

The final scenario is more end-to-end, with specific requirements: “You need to expose pre-computed analytics to your operational web application. How would you architect this?”

This is where things get interesting, as there are again multiple solutions to the problem.

Some examples of key questions :

• “Can you connect your data warehouse directly to your web app? What’s the usual latency ?”

• “What happens when 10,000 users hit your analytics dashboard simultaneously?”

• “Where would you place caching layers?”

• “How do you handle data freshness vs. performance trade-offs?”

The best candidates recognize this as a classic OLTP vs. OLAP problem and discuss solutions like read replicas, caching strategies, pre-aggregations and more.

What to look for

Both breadth and depth are important, and the shape of them will really depend on the role, as they are many different data engineer profile

Here are a few examples of the indicators I’m watching for.

Depth indicators

When someone truly understands a technology, they can discuss:

• Internal mechanics: How does Spark actually distribute work? What’s happening during a shuffle?

• Failure modes: What breaks when you lose a Kafka broker? How does Delta Lake handle concurrent writes?

• Performance implications: Why does a broadcast join beat a shuffle join for small tables?

• Operational realities: How do you debug a slow dbt model? What metrics indicate a streaming backlog?

Breadth indicators

Strong candidates can also:

• Connect different technologies and explain their relationships

• Understand why certain tools exist (what problem were they solving?)

• Recognize patterns across different solutions

• Make appropriate trade-offs based on constraints

Each level reveals more understanding. A junior engineer might stop at “Parquet is columnar.” A senior engineer discusses compression algorithms, predicate pushdown, and why you might choose Avro for streaming despite Parquet’s advantages.

Red Flags and Green Flags

How to prepare?

If you’re preparing for interviews like mine, here’s a few tips by order of importance :

1. Pick 3-4 technologies you use and go deep : understand their internals, failure modes, and alternatives

2. Practice explaining : If you can’t explain it simply, you don’t understand it fully

3. Build something real : even a toy Kafka setup teaches more than reading docs

4. Break things : Kill processes, corrupt data, fill up disks. Understanding failure teaches resilience

5. Read post-mortems : Learn from others’ production disasters

The value beyond AI

This interview style works because it mirrors real engineering. We start with vague requirements, ask questions, make trade-offs, and solve problems collaboratively. It’s rare for candidates to completely fail: everyone can draw something and explain at least some components.

What varies is depth, breadth, and the ability to reason about trade-offs. That’s what separates great engineers from good ones, and it’s what this interview approach reveals.

Remember: in the age of AI-assisted development, your value isn’t in memorizing syntax or configuration details. It’s in understanding systems deeply enough to design, debug, and evolve them.

Prompt engineering matters, but the fundamentals remain and will help you grow in this wild west era.

Events

• Low-Key Tech Data Happy Hour - Denver

• How To Build Right-Time Data Pipelines

Articles Worth Reading

There are thousands of new articles posted daily all over the web! I have spent a lot of time sifting through some of these articles as well as TechCrunch and companies tech blog and wanted to share some of my favorites!

The unwritten rules of being a head of data

A friend of mine became head of data but they had a hard time stepping away from coding. They kept shipping code late at night after their normal work ended.

This was fine at first until they realized, months later, they hadn’t really led anything of note for the business. There was nothing they had really driven.

They had been promoted, but still were working like a senior engineer. Delivering lots of well-designed work but not actually taking a stand on any specific project.

Once you become a data leader, the game changes. What the business needs from you isn’t just to write another Python script or notebook. But that’s not always made clear. Some companies or mentors will help guide you as you become a leader. Others will give you the promotion and then hope you figure it out.

In this article, I wanted to discuss some of the unspoken rules and lessons you need to pick up fast if you are a data leader.

Read More Here

Organization Architecture

By

The reliability of a service is directly impacted by the shape of the organization providing it. The shape, not the budget, headcount, or maturity level —as irresponsible leaders like to frame it!

This article elaborates:

• Why does the shape of the organization matter for reliability?

• How does it impact communication between teams and by extension interactions between the components of the system?

• What insight can be unlocked with consumer journeys to create a healthy organization that builds reliable systems?

Read More Here

End Of Day 204

Thanks for checking out our community. We put out 4-5 Newsletters a month discussing data, tech, and start-ups.

If you enjoyed it, consider liking, sharing and helping this newsletter grow.

Share

It might be that a business executive just came from a conference or read an article and now they are suddenly requesting your team to build an AI-powered, virtual 3-D pie chart that allows for self-service, dynamic drill-downs “like that demo they saw on stage.”

Or maybe someone from finance forwards you a Gartner report and asks why you still don’t have a Data Mesh, a Lakehouse, and whatever the latest marketing term is this month.

None of this is malicious...ok, perhaps a little from the marketing and sales from vendors…

It’s a symptom of a bigger problem:

Most of the vocabulary we use in data has escaped into the business without the underlying meaning.

People hear terms, “real-time,” “semantic layer,” “self-service,” “data quality”, and either don’t clarify their understanding or are waiting for someone else to properly define them.

This gap creates:

• Mismatched expectations

• Projects that sound good but don’t solve real problems

• And a lot of unnecessary fire drills

So in this article, I want to break down a handful of terms that data teams constantly find themselves explaining, not because the business isn’t smart, but because these concepts are overloaded, over-marketed, and often misused.

Batch And Real-Time - What Do You Really Need?

Read more

Today I wanted to discuss spreadsheets and well Excel. It’s a tool that continues to be used decades later by both non-data and data teams alike. For some, this is a frustrating fact as they think everything should be in code and version control. For others they are making it part of their workflows.

But before diving into today’s newsletter, I want to take a moment to thank this issue’s sponsor: Hex. Hex brings the magic of AI to data analysis workflows, whether you’re using code or no-code. Hex helps organizations work together with data and avoid jumping between different data tools for querying, data science, visualization, and spreadsheets. Over 1600 organizations use Hex to do everything from deep analysis to self-serve.

Now let’s jump into the article!

Can we get an Excel export?

For many data teams, this ask can be somewhat frustrating. You’ve likely poured hours or days into a dashboard only to feel as if the team that asked for it just wanted an Excel report.

Then, of course, there are all the one-off actual Excel reports and lookup tables being managed by a single person in companies. In fact, at many companies, it’s these one-off Excel reports, VBA scripts, and Google sheets that run whole departments from operations to accounting.

Not your polished, QAed, dynamic and impactful dashboard!

It really can feel like the meme below.

It might make many new and more experienced data professionals scratch their heads. We have all this technology, data platforms that can crunch more data than ever before, and yet, somehow, everything seems to continue to touch Excel.

I recently had one person show me the financial model they built in Excel using Claude. So, like the QWERTY keyboard, spreadsheets will be with us if we ever manage to set up a colony on Mars.

And like the QWERTY keyboard, there will always be a minority resistance group that complains about its inefficiencies.

But Excel and spreadsheets, for many companies, is the way they are data-driven. It is the way their execs make decisions.

With that, I wanted to discuss some of the realities of Excel and spreadsheets and where I see them going in the future.

Excel Is A Blackhole

It’s tempting to fight Excel after all, if everyone is just slicing and dicing data, it’s very easy to start getting inconsistent metrics, out of sync data sets, and just create confusion.

Even if you built the perfect data analytics platform with every data point checked, once it’s out Excel who knows what will happen to it.

But it’s where a lot of “self-service analytics” actually happens.

And here are a few reasons why:

• There Is A Low Floor And High Ceiling - Meaning, if you’re just getting started, you can use the SUM() function and slowly work up to VLOOKUPs, and for some people, you can take it to the point of building a rollercoaster simulator inside it.

• It’s Far Easier to Edit - If all I want to do is add $50,000 to a single field, it’s fast, easy, and doesn’t impact the permanent data source like you would if you were to try to do the same in your data warehouse.

• Instant What-If Analysis - Most BI tools aren’t really built for exploratory analysis. Excel? You can copy a tab, tweak three numbers, and immediately see the new forecast.

“Hey, what if our sales go up 12%… no, wait, 15%… actually, what if churn drops by 2%?”

• No Gatekeepers - In many large enterprises, if you go ask for data, ask for a dashboard, ask for new columns to be added to a data set. You get push back. So instead, you go to Excel, pull the data sets that you do have access to, add your own columns and build your own solution. Now you can get your job done without feeling like it’ll take six months.

Many of the reasons above are why both data analytics and governance teams often find Excel difficult to grapple with.

Who hasn’t had to deal with a one-off Excel lookup table that only a single person manages?

• Do you put it into a database table so the data team can now manage it?

• Do you want to be responsible every time there is a new field?

Maybe you’ll create a form or tool they can edit to manage the data in the table themselves(although that can be quite heavy).

And what about the other 100 Excel sheets that are doing something similar?

I suffered the same issue at Facebook, and for a while, we actually did have a tool that allowed me to build a pretty simple table app that allowed an end-user to essentially manage an isolated database table directly.

Which was great, I could control columns, data types, and even add in a few rules to check what data was being inserted while the end-user was working in a spreadsheet-like interface.

But eventually that got deprecated, and suddenly I was once again left with other teams’ Google sheets(this was after we started reducing our use of Quip).

So, the spreadsheets once again reclaimed their data.

Maybe Excel Is a Symptom of Deeper Issues

In many ways, I think it can be argued that if everyone is trying to circumvent your dashboards and go straight to Excel, that might be the signal for a larger issue. Here are just a few reasons they might be deciding to go straight to Excel:

• Your dashboards aren’t answering the real questions

• They need flexibility that dashboards don’t offer

• They don’t trust your source of truth(this would be a major issue)

• They can’t wait two sprints for a dashboard tweak

• They just want to build things themselves

• The dashboard is slow

• They feel safer making mistakes in Excel than in a tool they don’t fully understand

• They don’t want to watch your video on how to use the dashboard(or feel like they don’t have time)

I’ve been in situations where the marketing, finance, and operations teams use multiple dashboards to extract a few key datasets and then put them into their own spreadsheets. That should be a signal that the dashboards you’ve built may not be meeting their needs.

I think it’s important to understand these issues so you can:

1. Build better dashboards that can answer your end-users problems

2. Guide partner teams to better short and long-term solutions(if you know that your partner team is just

3. Prioritize fixes that actually unblock how teams make decisions

4. Set expectations on what BI tools are for and what they are not

5. Know when to build a more flexible interface instead of another static report

6. Spot patterns in how different teams work and design solutions that fit their realities

And I am sure you could still list more reasons. The truth is, there is more than one reason a team might decide not to use your dashboard or data product. You should figure out why to see if there are specific issues with the dashboard itself, or if there are more uses that the team needs for the data.

If You Can’t Beat Excel, Include It

Recently, several BI tools have leaned into spreadsheets. Which I believe is a smart move for them in terms of increasing the user base and meeting people where they are.

As many BI tools often require some level of SQL which sure you could ask everyone at your company to learn, but I don’t think it’s fair to expect(not unless data teams also understand the business better).

So seeing companies provide a layer of familiarly in the BI layer makes some sense. Maybe it’ll finally get us closer to actually being able to put governance around spreadsheets rather than fighting them…

I’ve seen this massively increase the number of users who interact with BI solutions because the users can finally do what they want.

• Write data back and manage ad-hoc lookup tables

• Create formulas and calculations as they do in spreadsheets

• Explore the data with the same freedom they’re used to, no rigid dashboard filters

• Build quick “what-if” scenarios without submitting tickets or waiting for dev cycles

• Annotate, comment, and explain numbers directly in the UI

I think most people would be surprised by what even a few small changes do in terms of making it easier to integrate workflows into your tool. Is it going to make data governance happy?

Probably not.

Attempts To Reduce The Chaos

Like I said earlier, we’ll likely never get rid of Excel or spreadsheets in general. It’s too convenient, too easy, solves immediate problems too well that if your goal as a data leader is to reduce the amount of spreadsheets that exist, its probably a fruitless effort. But I do think there are areas where analytics platforms can work to

• Detect Ad-hoc Usage Patterns - This was one of the projects I was considering prior to leaving Facebook. I wanted to be able to detect how end-users were using data to see if there were ways we needed to extend the current data platform before end-users asked us. Maybe there are tables being joined or custom columns being created. The challenge with Excel and spreadsheets is much of this happens outside of any form of tracking. But I’d still love to see Snowflake or Databricks integrate this into their platforms.

• Detect Metrics - This is one I’ve seen several tools and platforms offer. Where if they are seeing you define a metric, especially in a BI layer, they’ll suggest you create a universal metric and provide version control. I love the idea. After all, one of the challenges with some reporting and BI tools is they feel isolated. You create a metric in on dashboard and no other dashboard knows that it exists.

But none of this really brings Excel or spreadsheets under any form of governance or control. Not unless those spreadsheets exist inside some component of the data analytics platform itself.

Which is where I can see the world going. Especially if Snowflake and Databricks have their way and become the center for all your business data.

Excel - Villain, Symptom, or the Last Mile?

We often treat Excel like it’s the villain. Our nemesis in data.

In some cases, it can be. It can be a cause of many frustrating meetings where teams bring multiple conflicting metrics or maybe you offer to automate someone’s Excel process only to open up a 30 spreadsheet deep nightmare with more formulas than your VP has ‘quick asks.

But taking a step outside of the data team. For many, it’s their way of accessing data. Without anyone saying what they can and can’t do and you’re not going to take that away from them.

And, as the title suggests, many companies are run off Excel. So no matter how much some data teams want to put that genie back in it’s bottle, it’ll be around far longer than your dashboard.

As always, thanks for reading!

Events

• Low-Key Tech Data Happy Hour - Denver

• Right-time Data Integration for Snowflake

Articles Worth Reading

There are thousands of new articles posted daily all over the web! I have spent a lot of time sifting through some of these articles as well as TechCrunch and companies tech blog and wanted to share some of my favorites!

5 Things in Data Engineering That Still Hold True After 10 Years

When I started in the data world back in 2015 Hadoop was at it’s peak.

Actually I happened to be scrolling through an old instagram account and found a picture from a DAMA conference where Horton Works sponsored it(you can see it below). At the time, Hadoop and its ecosystem were everywhere, Hortonworks, Cloudera, MapR, each promising to reshape the future of data.

Fast forward just ten years and many newer practitioners don’t even recognize those names. And yet, in data engineering, a decade is barely enough time for fundamentals to change. Underneath the hype cycles and new logos, many of the same challenges remain.

Read More Here

Cash Rules Everything Around Me

By

I’ve been reflecting on the last week’s announced merger of two darlings of the data industry, and observing reactions to this news. In the practitioner community, it’s definitely a mix of Kumbaya and dread. Plenty of people are hopeful things will stay positive, but nearly everyone is making a Plan B (forking, budgeting, looking at new options, etc). I’m agnostic to whatever happens, and try to take a Zen-like attitude to these things, mostly because stuff like this is very common in our industry. It is what it is, and here’s some advice if you’re new to the game.

Read More Here

End Of Day 203

Thanks for checking out our community. We put out 4-5 Newsletters a month discussing data, tech, and start-ups.

If you enjoyed it, consider liking, sharing and helping this newsletter grow.

Share

Today’s article is an interview with Brian Femiano who is currently a staff data engineer at Apple.

He’s been doing data engineer for nearly 20 years with 9 of those as a staff eng. He’s worked across many domains including ad tech, music, social media, and intelligence. In his own words he enjoys mentoring tech professionals and helping folks level up, especially his managers. He lives with his wife Rachel and two boys.

Now let’s jump into the article!

Once you start pushing past senior-level, whether that’s as a data engineer, analyst or other IC, there are a lot of skills that start becoming important that have nothing to do with technology.

Or at least less so.

I’ve already written several takes on this in the past so I wanted to bring in someone else’s perspective. So for this article we have Brian Femiano who is currently a staff data engineer at Apple(in case you missed the intro) and he’ll be sharing about:

• Career Path And Motivation

• Defining The Staff Role

• Technical Design & Systems Thinking

• Collaboration & Communication

• Big-Picture Advice

If you’re stuck at senior or are just curious what it takes, then this article is for you!

Career Path & Motivation

Q. How did you get into data engineering? And what do you enjoy about it?

I started with Java services then one day got into the early Cloudera HFS/MapReduce training videos and got hooked. Distributed computing finally felt approachable for me and was really cool to see applied to data and not just number crunching.

Q. Can you share the key moments or projects that shaped your path from senior to staff data engineer?

Years ago I was on a team and someone needed to organize work into JIRA, assign that work out, document progress, help unblock teammates, and keep stakeholders outside of engineering informed. It’s not the kind of fun stuff devs enjoy doing but it had to get done. Once the org realized I had this holistic view of the project goals and how all the work could be organized to meet them, they recognized me as staff.

Defining the Staff Role

Q. How would you describe the difference between senior and staff data engineer in your own words?

It’s not an automatic flip so much as a gradual development. A senior dev might stick to a comfort zone and execute really well. The next step is non-coding skills that make other devs on their team better.

Things like:

• Diagramming systems and runbooks to help on-call

• Having good relationships with product folks

• Having a good relationship with other data teams to know how your changes impacts them.

Staffs also have to context switch more frequently in a given day.

Note From The Seattle Data Guy - I’ve always enjoyed the different Calendars on Will Larson’s website where he discusses many of the different archetypes of staff engineers. As Brian referenced about context switching, you might have to go from mentoring a junior engineer, to talking about scaling issues, to then discussing budget, interviewing, planning offsites, etc. Actually, in the next answer Brian will reference his thoughts against the archetypes idea. Which I do understand. I could write another article about my thoughts there, but for now let’s go back to Brian!

Q. What misconceptions do people have about the staff title?

• That you need to be the most talented developer on your team

• That everyone fits into these nice little archetype roles like classes in an RPG video game.

• That job hoping is the best way to get the title.

Q. What do you find is the most common blocker for senior data engineers trying to get to staff ?

Poor or non-existent communication or being abrasive. Not flexible based on circumstances or able to see the big picture of their work. Being too focused on tools/languages and just generally not building for long term system health and quality.

Q. For a senior engineer aspiring to staff, what specific signals should they show?

There’s a ton but I’ll rattle a few off.

That their managers and peers know they think holistically about the systems you’re building. Thoughts are put into diagrams and accept feedback well.

Support your teammates during fires.

Generally you want to make yourself indispensable to your team but not in a way where you have siloed knowledge or also always having to play the hero.

It’s a fine line to walk.

Technical Design & Systems Thinking

Q. When faced with a complex request, what’s your process for designing a simple, scalable solution?

Immediately diagram my thoughts.

What currently exists that might help satisfy the request?

Are we happy with those systems or is this our chance to refactor problem areas?

What’s the minimalist set of new capabilities needed to be built?

Can we leverage existing libraries?

Are any parts of the design susceptible to bottlenecks as load increases?

Does any part of this need constant manual attention or is it mostly automated?

Q. How do you decide what to build yourself versus what to delegate or break into
components?

This is a tough one. The team leads I learned from tried not to involve themselves on the critical path. I’ve broken this rule many times though based on circumstances. If you assign yourself too much it can delay the project and also deprive your teammates of growth, both of which hurts the business. Generally you want to trust the most important areas to your teammates. Figure out where you can pitch in to help avoid overload.

Q. Can you share an example of a “simple” solution to a complex problem that you’re
particularly proud of?

Back at Pandora we wanted a service that would notify artists when one of their songs was added to certain DJ curated playlists.

The business objective was clear: If we send them sharable links, then they might post them organically and give us free publicity.

We started out thinking we’d fire emails in real time for any new Kafka events, but this had a number of unattractive tradeoffs.

After some back and forth with product we realized artists didn’t even want the notifications in real time. We ending up building a minimal set of reliable components that read the events and batch the emails.

It’s still in prod today.

Collaboration & Communication

Q. What have you learned about keeping managers, product partners, and other engineers aligned on big projects?

With management I try to be succinct and timely so they can stay informed on progress with minimal effort. For product it’s a lot of asking questions and note taking. Repeating back things they tell me in my own words. Engineering it’s about giving as much detail and clarity, even if it means repeating yourself many times. Exercising patience with everyone. I’m still working on all
of this.

Q. How do you create an environment where teammates feel comfortable approaching you with questions or concerns?

I think it starts with just being friendly and non-judgmental so they can ask me anything. I try to prioritize anyone who’s asked for my help to unblock them. Also give your teammates the leverage when they speak up and have a good idea you didn’t think of.

Q. Any tips for writing design docs that earn trust and reduce confusion?

Start out with 3-5 sentences outlining what’s being built and what are the business benefits. Reference other parties involved in the project so readers know everyone involved and can reach out themselves. Find a diagramming tool you like and focus on diagrams that communicate how pieces fit together. It’s less about how good the diagrams look artistically and more about how easy they are to follow.

Big-Picture Advice

Q. Where do you see data engineering going over the next few years. Anything you’re most excited about?

We’re already starting to see data engineer roles looking for proficiency in languages beyond just Java/Scala/Python/SQL and that will continue. It’s also been cool to see orgs recognizing that their problems aren’t with volume but more governance and data quality, and choosing tools/platforms more aligned with that reality.
It’s exciting to see some of the open table formats evolve and help teams with those areas. I also have a somewhat optimistic take on GenAI in this space. There’s so much talk of it replacing entry level roles but I don’t think so. The entry level folks I interact with are the ones really good with it and schooling the seniors.

Note From Seattle Data Guy - If you’d like to learn more about Brian and his past work, you can also check out the video below!

Final Thoughts

This is Ben Rogojan again!

What a senior and staff engineer is means different things at different companies(actually I guess I could say this is even true amongst teams). I’ve seen some be more focused on architecture, others coding machines and still others play a mix of a technical project manager, strategic partner to the business and mentor to more junior engineer.

The through line is that these engineers have some form of outsized impact. We all only have so many hours in the day and we can only code so fast. At a certain point we have to start considering more about what we are coding vs. just the speed.

For example:

• Finding and working on projects that move the business forward

• Helping the business remove ambiguity

• Keeping the business from picking a solution that will cost them millions and require them to migrate in 18 months.

I am so glad Brian shared his experiences. I hope you found them valuable!

Events

• Low-Key Tech Data Happy Hour - Denver

• How To Build Right-Time Data Pipelines

Articles Worth Reading

There are thousands of new articles posted daily all over the web! I have spent a lot of time sifting through some of these articles as well as TechCrunch and companies tech blog and wanted to share some of my favorites!

Beyond Big Tech: The Reality Of Data Engineering Outside Silicon Valley

I’ve had the privilege of working at a mix of companies—big tech, start-ups, enterprises, and nonprofits. Each has given me a unique perspective on how different environments approach data: the infrastructure they build to manage it and how they ultimately use it.

I’m writing this because, while tinkering on a small side project, I was hit with a vivid reminder of just how frustrating it can be to work with data from certain industries. In fact, when I made a humorous post about it, someone called out the poor data model in the raw file I was using.

But that’s the thing: it wasn’t my model, that’s just how the data came.

Read More Here

Building Zone Failure Resilience in Apache Pinot™ at Uber

Initially, our Pinot clusters at Uber relied on two key strategies: tag-based instance assignment, which groups servers by tenant to ensure logical isolation, and balanced segment assignment, which spreads data segments evenly across those servers. While effective for distributing data evenly across servers within a tenant, this approach didn’t inherently guarantee distribution across different physical zones. If all instances assigned to a table, or all replicas of a segment, happened to be in a single zone, a failure in that zone would lead to significant service disruption.

Read More Here

End Of Day 202

Thanks for checking out our community. We put out 4-5 Newsletters a month discussing data, tech, and start-ups.

If you enjoyed it, consider liking, sharing and helping this newsletter grow.

Share

Before diving into today’s newsletter, I want to take a moment to thank this issue’s sponsor: Hex. Hex brings the magic of AI to data analysis workflows, whether you’re using code or no-code. Hex helps organizations work together with data and avoid jumping between different data tools for querying, data science, visualization, and spreadsheets. Over 1600 organizations use Hex to do everything from deep analysis to self-serve.

Now let’s jump into the article!

A few months back, I wrote an article about what hasn’t changed in the data world. And much of what hasn’t changed are the problems we face.

Of course, there are also plenty of things that have changed in the data world since I started. For example, the technologies and practices we use.

Even the words and terms we use, although mostly the same, have changed. Whether you like it or not.

When I started, no one used the term analytics engineer, and even the concept of a data engineer was still relatively new(at least in terms of how popular it later became).

In the same way, I have seen plenty of things change. Some of these changes are temporary, I believe (like my first point), others are likely larger trends.

So let’s dive into what’s changed in the data world in the last decade.

1) Everyone Wants Seniors On Their Team

Something strange started to happen a a few years ago. Whenever I was speaking with data leaders and they brought up hiring, they always referenced senior engineers and analysts. That’s the only position they were looking for.

It didn’t really click what was going on until the chart above came out recently…and well things became clear.

Both in terms of anecdotally and in terms of data there is a trend to hire more senior engineers over a mixed team.

I’ve heard a couple of causes:

• A senior engineer with a copilot can get far more work done by themselves.

• Senior engineers tend to be able to jump into problems faster and apply what they’ve used in the past to solve said problem.

• Data teams are smaller than they were a few years back and in turn they don’t have the head count for juniors.

That last point is from my experience, so it’s anecdotal. But, I think it’s also what many other consultants and data leaders are seeing. Smaller teams, perhaps in response to the considerably larger data teams of the early 2020s.

I do have some friends who are currently working to try to create tools to help junior engineers become more productive even if a senior engineer is around. And I imagine there will be a correction in this trend. Because if you only hire senior engineers, eventually said seniors will just demand more income until it’s no longer viewed as cost effective to mostly hire seniors.

Don’t just take my word for it, feel free to share your own thoughts.

Leave a comment

2) Let’s Just Start With The Cloud

It was only a few years ago where the cloud was still an option that companies would consider.

It wasn’t the default.

Now I find many companies and data teams jump to the cloud immediately. And those not on the cloud already are migrating. Over the past few years I’ve done over half a dozen SQL Server to Snowflake migrations. Don’t just take my word for it. Looking at the chart above from Accenture back in 2021 also showed a similar trend of migrating away from on-prem to the cloud for analytics.

Cloud has become the first choice for many companies.

This is true up and down the ladder in terms of company size. Small, large, medium, I haven’t had one project where someone decided to run a database or data pipeline locally. Other than to test an idea or via a dev workflow. I know, some of y’all are probably reading this thinking, well, I am still on-prem!

There are still plenty of data stacks on-prem. Some have to be for regulations others find it cost effective to be on-prem.

However, in my experience, I am running into fewer and fewer teams that are relying on an on-prem solution for analytics.

3) Data Teams Don’t Jump To Building Their Own Custom Data Pipeline Solutions

When I first started in the data world, many “data pipeline” systems were built with cron, Windows task scheduler, maybe SQL Server agent as the scheduler, calling either a bash or Python script, or directly calling SQL or SSIS.

Most companies I’d walk into would likely have built their own data pipeline and orchestration solution. Many of which were built over the past decade or so, and you’d literally be able to sense when data leaders changed because the style of how things were written would change.

Now this isn’t the default.

Instead, there are plenty of other options, especially with the Cloud(again), from as simple as using a Lambda and Eventbridge or going straight to a tool like Airflow. People are using pre-built tools to manage their automated pipelines.

I imagine there are many reasons for this:

1. There is an expectation to deliver actual impact faster. Meaning if you spend time just building a data orchestration system which requires you build logging, meta-data tracking, scheduling and other components before even delivering a data pipeline, you’re likely going to hear it from the business.

2. There is a lot more intro content out there, and a lot of it starts with tools such as Airflow. So if you’re a new data engineer or looking to set-up your first pipeline for your company, you’ll likely run into it and start there.

3. Job requirements often ask for tools such as Airflow, Coalesce, dbt, and so on which means that’s what new data engineers pick-up.

Whatever the reason, building heavily custom data pipeline systems seems to be a less frequent occurrence.

As always, I’d love to hear your thoughts on this as well.

4) No One Is Questioning SQL As The Lingua Franca Of Data Anymore

Alright, that’s not 100% true. There are still people who dislike or want to replace SQL. In fact, I usually find that most data engineers fall into liking. Either SQL or Spark(via data frames and Python).

However, had you joined the data world in 2015, every tool seemed to be building its own query language. Some were just offshoots of SQL, like BigQuery’s Legacy SQL, HiveQL, etc.

Other query languages tried to build a completely new approach to how you queried data. And of course, there was NoSQL, which was always touted as “Not Only SQL. But it’s hard for me not to read it as a way of pushing against SQL in both an underhanded/overt way via a name.

Regardless, here we are in 2025, SQL is more popular than ever.

You can check out Luke Barousse’s graph above, where he scraped data for job requirements. As you can see, SQL is almost always required. People now start by learning tools that further enforce that, like dbt.

5) AI Is Changing Workflows

It’s impossible not to acknowledge that AI is playing a role in how data teams work. Whether someone is just adding in an LLM to get rid of having to write DDL statements, or your IDE has three different copilots running at once. AI is changing the way we work.

On the positive side, I’d say I’ve personally used it and seen it remove a lot of the redundant work I’ve done in the past.

On the more negative side, I’ve also seen people start to rely on it so heavily that they don’t really look into debugging. Even when an error arises, they just put said error back into their copilot and have it try to figure out the problem. Sometimes this works, but many times I’ve seen more and more code bloat get written as a result…I guess we kind of debugged in a similar fashion during the peak of Stack Overflow.

My final thought here is that: I wouldn’t be surprised if some of that feeling of efficiency we get when using an LLM is more similar to just the feeling of movement over progress.

Final Thoughts

The more things change, the more they stay the same. But things do change. Not overnight. Just because someone in SF released some magic box that makes us all constantly question our own self-worth, doesn’t mean that tomorrow all work will cease.

Just because many of our data workflows are online, we will magically start finding new value from data, no matter how much marketing there is out there.

What I have seen change is the size of companies looking into their data, and the speed at which companies are demanding data to provide them value. Perhaps that’d better be rephrased as the impatience the business has for the data team. This could account for the hiring of seniors and the usage of the cloud, which might be viewed as faster.

But don’t let me be the only one sharing. I’d love to hear your thoughts, what has changed, what has stayed the same?

Feel free to comment, and as always, thanks for reading!

Articles Worth Reading

There are thousands of new articles posted daily all over the web! I have spent a lot of time sifting through some of these articles as well as TechCrunch and companies tech blog and wanted to share some of my favorites!

How and Why Netflix Built a Real-Time Distributed Graph: Part 1 — Ingesting and Processing Data Streams at Internet Scale

The Netflix product experience historically consisted of a single core offering: streaming video on demand. Our members logged into the app, browsed, and watched titles such as Stranger Things, Squid Game, and Bridgerton. Although this is still the core of our product, our business has changed significantly over the last few years. For example, we introduced ad-supported plans, live programming events (e.g., Jake Paul vs. Mike Tyson and NFL Christmas Day Games), and mobile games as part of a Netflix subscription. This evolution of our business has created a new class of problems where we have to analyze member interactions with the app across different business verticals. Let’s walk through a simple example scenario…

Read More Here

Is “data-driven” just slowing down your decisions?

For decades, companies have been chasing the idea of being data-driven. I recall when I first started the term was plastered everywhere.

Every blog you read and conference you went to, someone referenced that term along with the idea of data being the new oil.

Everyone wanted to be driven by data. After all, companies like Facebook and Google were doing so well. And not just them, there were plenty of statistics that showed that companies that used data were more profitable than their competitors.

Read More Here

Is It Time to Say Goodbye to Data Engineers?

Ever since tools like SSIS came onto the scene, vendors and business leaders have been on a mission to remove what they see as the biggest roadblock to data-driven decision-making: data engineers.

Or their counterparts—DBAs, ETL Developers, and Data Architects.

Sure, not everyone says it so explicitly, but you can see it in vendor marketing and in the decisions made by the business.

I remember talking to a veteran data expert who’s been in the field for three decades. They told me that when SSIS first launched, people were genuinely afraid for their jobs. The idea that you could just drag-and-drop tasks that once required code was nerve-racking. But if you’ve used SSIS, well, you know the truth.

To some extent, I get why the idea is appealing. When a leader requests a report, a software engineer wants to modify an application table, or a data scientist wants to explore a new dataset, who’s the one slowing down the project?

The data engineers.

Read More Here

End Of Day 201

Thanks for checking out our community. We put out 4-5 Newsletters a month discussing data, tech, and start-ups.

If you enjoyed it, consider liking, sharing and helping this newsletter grow.

Share

Hi, fellow future and current Data Leaders; Ben here 👋

Today I’ll be reviewing some trends and thoughts I’ve been mulling over for 2025. This newsletter is also my 200th newsletter! So I’ll take a quick moment on that as well.

But before diving into today’s newsletter, I want to take a moment to thank this issue’s sponsor: Hex. Hex brings the magic of AI to data analysis workflows, whether you’re using code or no-code. Hex helps organizations work together with data and avoid jumping between different data tools for querying, data science, visualization, and spreadsheets. Over 1600 organizations use Hex to do everything from deep analysis to self-serve.

Now let’s jump into the article!

First off, woah, what a busy 2025!

It’s flown by and with that, I am also realizing that this is my 200th newsletter!

200!

Thank you to everyone who has been reading it really does mean a lot.

I had personally meant this final quarter of 2025 to be a bit of a breather and yet, but as they say…

If you want to make God laugh, tell Him about your plans.

I’ve been traveling and speaking.

Working with a lot of great clients.

A start-up I invested in got acquired(ok that didn’t really involve me). So many congrats to the team.

Another start-up I advise just raised their series A, which I’ll talk more about later.

Not to mention I went to Europe for the first time and actually was able to take some time off without panicking and needing to open my laptop every five minutes.

Somehow I still have more to do for 2025.

With all of that I did want to share a trends and points my mind keeps going back to as I am now starting the path to my next decade in data.

The Data World Spans Decades

If you open up Google and start looking into different ways you could set up your data infrastructure, you’ll likely find hundreds of possible designs, from Netflix and Uber to newsletters like mine.

It can be tempting to think that every company out there must be using the latest and greatest technology. But as someone who has seen dozens of companies, I can tell you firsthand that is not the case.

We are at a point where companies are in different decades of data infrastructure.

Some are using Snowflake, Airflow, and Coalesce.

While others are successfully using SSIS and SQL Server.

Still others are on Hadoop.

All of which can be functioning.

Would I recommend anyone go and try to spin up Hadoop now? No. But there are a lot of technologies still in play. So don’t expect every job you work to always be with the latest and greatest tools. At the end of the day, if your data team is delivering results, that’s what matters.

Until you need to interview…

The Data World Today Will Change Tomorrow

When I first started in the data world, there was a lot of talk about Data Lakes. Now that talk has slowed or converted into Data Lakehouses.

In 2020, there was a part of the data world that spoke heavily about the Modern Data Stack.

Now that has also gone.

Many of the tools you’re spending time learning today might not be the “it” tool in 5 years.

In fact, they probably won’t.

We’ll likely have figured out how to better integrate LLMs into workflows and where they fit best. A list of new tools with new names will start to replace our current options.

The reason I say this is that I’d often get questions on my videos about skills.

“Will this be important to know in 5 years?”

“Will this career be around for the next few decades?”

We work in tech. Things change.

I believe there will always be a need for individuals with technical capabilities.

People who are willing to solve hard problems. But some of those problems will likely look different in a decade, and you will have to learn new skills.

Flexibility as a Principle

One theme that is emerging is that of interoperability and flexibility. We’ve spent the last few decades often having to choose.

Streaming vs batch, Spark vs Presto?

Shouldn’t a tool just allow us to use what we need when we need it?

Shouldn’t tools also be able to talk to each other?

Many data tools are starting to provide this increased flexibility and I see this becoming more of a norm.

For example, Estuary, a company I’ve been working with for several years now, gives end-users the choice when it comes to when their data arrives. Do you want your data now or in batch?

If you need it streamed, do it, but if you don’t and you’re connected to Snowflake, so you want to keep your Snowflake costs lower, load it in larger chunks. Meaning you could load multiple sources with different patterns if that would benefit you.

You can deliver your data at the right time.

Similarly, there are many tools now that are positioning themselves as compute-agnostic. Just use Apache Iceberg and then pick which compute engine you want to use. Presto, DuckDB, etc. Then you can serve the data in Snowflake, Databricks, or just keep it in Iceberg.

On the interoperability side, it seems as if every company now wants to be known as open, whether they are or aren’t.

Final Thoughts

Once again, thank you to all the readers of this newsletter. Your support means so much!

Whether you’ve been here since issue #1 or joined somewhere along the way, writing these each week has been one of the most fulfilling parts of my career.

As we start winding down this year, it’s important to remember that change will happen. Technologies update and get replaced.

Tools and buzzwords that were popular today might fade into the background. But I do believe we will still need individuals who enjoy tinkering, testing, and solving problems.

With that, I want to say here’s to another 200 newsletters, and thanks, as always, for reading.

Upcoming Data Events I’ll Be At

• Queries, Cocktails & Community - Small Data SF Happy Hour

• Small Data SF

Articles Worth Reading

There are thousands of new articles posted daily all over the web! I have spent a lot of time sifting through some of these articles as well as TechCrunch and companies tech blog and wanted to share some of my favorites!

Anyone Else Struggling to Keep Up With Data Tools

Are you managing to keep up?

One of the truths I’ve realized about working in the data world, and really the technology world, is that you can work a lifetime and never ever touch massive swaths of technologies.

Like Sisyphus, you could spend every day pushing the boulder of new technology up the hill, only to have it roll down back to the bottom.

You could only work on Snowflake or Databricks, and in twenty years, never have touched the other. Okay, that might be unlikely, given how some companies set up their data stacks. Somehow, even smaller companies are finding a way to use both. Probably some classic vendor-driven development.

Still, if you’ve only worked in big tech, you might never see some of these solutions while working on unique tools that are developed in-house.

So how do you actually keep up?

Read More Here

DataViz 101: Key Principles for Crafting Clear Dashboards

By

I’ve reviewed more than 3,000 students’ dashboards over the past three years, and here are my tips if you’re just starting your data visualization career.

Read More Here

End Of Day 200

Thanks for checking out our community. We put out 4-5 Newsletters a month discussing data, tech, and start-ups.

If you enjoyed it, consider liking, sharing and helping this newsletter grow.

Share

Before diving into today’s newsletter, I want to take a moment to thank this issue’s sponsor: Hex. Hex brings the magic of AI to data analysis workflows, whether you’re using code or no-code. Hex helps organizations work together with data and avoid jumping between different data tools for querying, data science, visualization, and spreadsheets. Over 1600 organizations use Hex to do everything from deep analysis to self-serve.

Now let’s jump into the article!

Early in my career I had a conversation with a friend who was working as a director leading both software and data teams.

One of the points they brought up was the fact that they often found that some of their senior level analysts and data scientists would provide them massive chart dumps when they ran any form of analysis.

Somewhere deep in a word document or python notebook layered with charts and explanations might be a conclusion.

I had actually not considered this point before having this conversation.

Now it’s been years since that chat and since then I’ve had nearly a hundred conversations with data leaders, and several skills and traits kept standing out in terms of what differentiates engineers and analysts. Here are three of them.

Getting Buy-In And Influencing Decisions

I don’t recall who told me this idea before, but someone once described that at every level(junior, mid, senior, etc), there are levels inside it. This breaks down into more “junior” and “senior” at that specific level.

Once you start going from a more “junior senior” to a “senior senior” many data leaders I’ve spoken with are looking to see if their ICs can start to get buy-in for their projects.

At this level many senior engineers and analysts are likely starting to feel antsy. Like they should be promoted to staff. And although every company has a different definition of what staff means, having the ability to get buy-in came up several times as a differentiator.

This is hard because it involves a combination of skills and traits.

• You need to be able to communicate an idea

• You need to have built some credibility in the organization

• You need to be able to understand the needs of other teams

Each of these aspects require time. But, you don’t have to start with some massive project.

How to Build Buy-In as a Senior IC

1. Start Small – Pick a recurring pain point, propose a lightweight fix, and see it through. If you’ve never got buy-in for a small idea, you won’t be used to what push back you’ll get nor will you have the credibility to lean on.

2. Draft a short one-pager - This should outline the problem, impact, and a lightweight solution. We often think other people are in our heads and get what we mean when we say it. Writing things down often highlight where teams might not be aligned or where issues will arise. So do it and figure out how to resolve said issues.

3. Socialize Early – Float ideas in 1:1s before you “pitch.” I had plenty of ideas get shot down because I never asked individuals around the key stakeholder what they might think of the pitch. Nor did I bring anyone else in on the idea so they could feel they had helped shape the idea.

4. Map Stakeholders – Know who’s affected and what each group values. There will always be push back, especially when you start asking for larger requests. Prepare for it based on the various stakeholders.

5. Deliver Consistently – Reliability is credibility. At the end of the day, if you’re not delivering, all the pre-work won’t matter.

6. Show Impact – Translate technical work into business outcomes, time saved, ROI. Don’t just talk about the technical improvements as this likely won’t land.

Read more

Before we dive in, a quick plug for Retool. Their new AI-assisted dev capability lets you build internal tools on top of your live data—without wrangling a prototype or settling for a static BI dashboard. Describe what you need, and Retool creates a secure app you can fully customize. Perfect for building interactive dashboards and admin tools connected to your warehouse, dbt, or orchestration metadata.

Now let’s jump into the article!

It’s the early 1950s, and an American journalist named Edward Hunter begins publishing a series of reports(and eventually books) about U.S. prisoners of war in Korea. He claimed they were being “brainwashed”.

But the term brainwashed up until this point actually didn’t exist in English (or at least wasn’t popularized).

The term caught fire. It appeared in headlines, films, and congressional hearings. It’s important to note that Hunter wasn’t merely a journalist; he was a seasoned propaganda expert who had served in the OSS, the predecessor to the CIA.

The U.S. military seized on the idea of “brainwashing” to discredit confessions made by American POWs, including statements admitting to biological warfare. Depending on the source, some claimed that the word “brainwashing” was injected into the English language solely so the military could have a defense against the American POWs who were making said statements.

Now, why am I writing about this?

The Seattle Data Guy who now lives in Denver?

Because many of the words we use in our day-to-day as data engineers and analysts were only created in the past few decades, and yet many of them dictate our choices on tooling, on design, and the work we take on. So I wanted to dig into how the words that we use play that role.

Words Shape the Way We Work

I used brainwashing as an example because it’s a little dramatic, and it shows how language can shape how we see the world, how it can excuse behavior or choices because we now have a word to define it. But not every term is born from manipulation. Sometimes, we’re just trying to name something new we’ve noticed, like a pattern or behavior.

And when you do it well, when you manage to wrap something complex into a simple phrase, the word will spread. It creates a shared language. It makes it easier for others to build on your thinking and to bring structure to what was previously vague.

So I don’t view this as inherently bad or negative. But it can limit our perspective on the world. These words and terms are also often self-replicating, passed from one company or organization to another long after their meaning has drifted.

It All Started With The Data Warehouse

It’s hard not to start this dive into terms that have shaped the data world without mentioning data warehousing. It remains a central part of our work today.

It’s why you’re stuck on the migration project from SQL Server to Snowflake.

It’s why everything seems to revolve around lakes and lake houses. It’s all merely an evolution or variation of that term. There might be a better approach, but why question it?

Speaking of the Lakehouse.

I’d be remiss not to reference the Data Lakehouse. I think you could argue that this one concept alone(and Databricks putting everything they could behind it) willed Databricks into the spotlight as more than just a managed Spark service, even though it has been difficult to shake that label even to this day.

It’s funny because Databricks wasn’t even the first company to use the term. But Databricks did everything they could to legitimize the term even further. The first article around the topic was around January 2020, and then somewhere in early 2021, the idea took off(see Google Trends below).

Again, I do think this was capturing an idea that had already been floating around. It seemed like a natural combination of Data Warehouse and Data Lake, which is why I believe the idea was so potent.

We were tired of having to build both.

Snowflake was likely very aware that this term gave Databricks a new mental road in people’s minds. This was an opportunity for Databricks to start taking workloads from Snowflake. Thus, much of Snowflake’s marketing opted for the term “Data Cloud”.

In fact, I recall a video I recorded with a Snowflake employee that was never aired. I mentioned Snowflake as a data warehouse, but the employee insisted I re-record it and never let that video see the light of day.

Personally, I think Data Cloud lacks a certain level of graspability. It really is meaningless.

If you’re creating a term, it needs to capture your audience effectively. If it’s so broad that it could literally mean anything(as I believe data cloud does), it probably won’t catch on.

Death Of Terms

Some words and concepts last for decades.

Others barely make it a decade.

When I first started in the data world, various vendors were pitching schema-on-read, and not too long after that, the term Modern Data Stack came around.

Perhaps one of these terms was created with limited longevity in mind. I say that mostly in jest. I think most companies have no idea what terminology will stick. Maybe that’s why Databricks keeps trying to invent new terms.

But like some viruses, a population might slowly become inoculated to an idea. Perhaps due to overexposure or due to the vaccine of reality, people start to question it. Break it down and push it aside. Once enough of a population has rejected an idea, it slowly ceases to exist.

That’s what has seemed to happen to both schema-on-read and now Modern Data Stack. Well, for many, the idea of the MDS passed away a while back.

Final Thoughts

I write this article both for myself and for everyone reading. The terms we use in the data world are useful and can help us have a shared language.

They can also trap us in our current world, where we limit our ability to see beyond the walls of our own minds.

Those ideas can be hijacked by vendors to sell their products.

They can also play naturally in a space that was always looking to be better defined.

We’ve had so many terms floated in the past few years that it can feel overwhelming.

And I bet that shortly we’ll have some new terms that redefine an all-in-one data stack coming out real soon.

If you’re just getting started in your data career, I owe you an article with a longer breakdown of all the terms and their timeline almost like a vaccine to ideas.

As always, thanks for reading.

Upcoming Data Events I’ll Be At

• Queries, Cocktails & Community - Small Data SF Happy Hour

• Small Data SF

Articles Worth Reading

There are thousands of new articles posted daily all over the web! I have spent a lot of time sifting through some of these articles as well as TechCrunch and companies tech blog and wanted to share some of my favorites!

Why is everything so scalable?

I’m entirely convinced that basically every developer alive today heard the adage “dress for the job you want, not the job you have” and figured that, since they always wear jeans and a t-shirt anyway, they might as well apply it to their systems’ architecture. This explains why the stack of every single company I’ve seen is invariably AWS/GCP with at least thirty microservices (how else will you keep the code tidy?), a distributed datastore that charges per query but whose reads depend on how long it’s been since the last write, a convoluted orchestrator to make sure that you never know which actual computer your code runs on, autoscaling so random midnight breakages ensure you don’t get too complacent with your sleep schedule, and exactly two customers (well, potential customers).

Read More Here

If You’re New to Data, Read This Before You Build Anything

When you’re just getting started in data, everything feels exciting, and everything sounds like a good idea.

“Oh, this process takes ten minutes? I’ll automate it with VBA!”

Fast-forward four weeks, and you’re trying to meet the finance team’s expectations, reconciling numbers, and banging your head against a table, wondering why you ever volunteered.

New paradigms show up with shiny names and polished diagrams. They sound smart. You’ve got nothing to compare them to, so you try them. After all, everyone else seems to be.

That’s what this article is about: the things I wish I understood earlier in my data career. It started because there were so many buzzwords and new products popping up in the past few weeks, it might not be clear what’s actually going on if you’re new to the data space.

Whether you’re early in your data career or just want a sanity check, here’s a breakdown of the stuff that actually matters (and the fluff that doesn’t).

Let’s dive in.

Read More Here

End Of Day 198

Thanks for checking out our community. We put out 4-5 Newsletters a month discussing data, tech, and start-ups.

If you enjoyed it, consider liking, sharing and helping this newsletter grow.

Share

Before diving into today’s newsletter, I want to take a moment to thank this issue’s sponsor, BMC. Many data teams still spend too much time babysitting pipelines and stitching together scripts. Control-M helps automate that work, coordinating data workflows, triggering ML jobs, and syncing across cloud platforms, so teams can focus on analysis instead of maintenance.

Check out their latest case study to see how it can simplify complex workflows.

Now let’s jump into the article!

A common challenge for data teams is figuring out what to work on that will drive the most value.

In some cases, the business has projects they believe should be taken on that you might disagree with, but you don’t have a solid list of suggestions, so you’re stuck with what they’ve asked you to work on.

In other cases, the business team might not know exactly how they’d like to use data, other than perhaps a dashboard or two. That means you can’t just ask, “What data projects do you need?” and expect a helpful answer.

Instead, you need to ask questions that uncover pain points, opportunities, and processes where data could make a meaningful difference.

At Facebook, we conducted this exercise every six months to realign priorities and identify new, high-impact projects. We’d go and talk to our business partners to understand their current and future needs. From there, we’d work to come up with projects we believed aligned.

If you are looking for questions to help you better align with the business, here are seven questions you can ask to surface valuable data. (Also, I’d love to hear if you have any questions you’d recommend.)

1) What’s keeping your team from hitting its targets right now?

Not everything is about data. Many leaders recognize that there may be areas where they are struggling to meet targets.

Perhaps they aren’t converting enough on ads, or their team has consistently missed the mark when it comes to delivering products on time. There are real-life situations where data may or may not be helpful.

But instead of asking, Where can data help, your focus should be on understanding the bigger picture.

Department Specific Versions Of This Question

• Finance - Are there recurring surprises in expenses or revenue that you wish you could predict earlier?

• Marketing - Are there channels where you feel you’re spending too much but not seeing results?

• Operations - Where do delays or inefficiencies tend to pop up in your daily workflow?

Read more

Before diving into today’s newsletter, I want to take a moment to thank this issue’s sponsor, BMC. Many data teams still spend too much time babysitting pipelines and stitching together scripts. Control-M helps automate that work, coordinating data workflows, triggering ML jobs, and syncing across cloud platforms, so teams can focus on analysis instead of maintenance.

Check out their latest case study to see how it can simplify complex workflows.

Now let’s jump into the article!

I’ve been think a lot about the fact that for many companies where the data team work ends, the business value only starts to begin. Even with a simple dashboard or model. You deliver the data product, then what?

Value is supposed to appear right?

After all that’s what the speaker said at a conference “data teams need to provide value” or some other cliche line.

Of course data teams need to provide value. But how a team provides value is often left vague.

That’s where many leaders get stuck. They want their team to be “strategic” or “business-driven,” but they haven’t clearly defined what role their data team actually plays.

A team tasked with building out a data warehouse will provide value in a very different way than a team embedded with marketing to improve campaign ROI.

Over the years, I’ve seen multiple types of data teams. It can be tempting to think about data teams based on roles. That’s the analytics team, the data engineering team, the data infra team, etc. But I think it’s more important to think about the function you want your data team to have.

What are the goals, what is the business function it provides.

Sometimes data teams are strategic partners who connect business problems to data solutions. Others might be enabling other teams by being the platform builders who develop and maintain the data infrastructure and warehouse.

And in some organizations, the data team is just 1-2 people who have to do all of the above.

To figure out how your team provides value, you first need to understand the role it plays for the business.

Four Ways Data Teams Provide Value

A framework I enjoy when considering team types comes from Team Topologies: Organizing Business and Technology Teams for Fast Flow. While the book is about technology teams broadly, its patterns apply neatly to data teams too.

I’ve listed a few examples below.

Platform Teams

Platform teams are often a central infrastructure team that builds layers that everyone else can build on top of. For example a data engineering team that builds ingestion pipelines and manages the warehouse. Their value comes from enabling all other teams to access clean, reliable data. And if you’re in a large enough organization the data engineering team might be building on another platform team, the data infrastructure team.

It is important to note that a company only benefits from having a large investment in platform teams when there are enough analysts and end-users actually building tools and data products that the business uses.

Complicated Subsystem Teams

Sometimes companies need specialists. That’s where these teams come in. Think of a data science research team focused on building recommendation systems or a supply chain optimization model. Their value comes from delivering deep expertise that directly improves the business by either improving operating efficiencies or increasing revenue.

Stream-Aligned Teams

These are domain-embedded teams that work directly alongside a specific business function, like Marketing, Operations, or Product, to deliver data products that drive day-to-day decisions. For example, a marketing analytics team embedded within the growth organization that builds dashboards, models, and experiments to directly influence campaign spend and customer acquisition.

Their value comes from proximity, being close enough to the business to understand context, move quickly, and measure impact. They turn data into decisions without layers of translation between business and data teams.

Enabling Teams

This would be some form of Data Enablement Team within a larger analytics organization. They don’t own production pipelines or dashboards, instead, they help other teams become more capable with data. At least, how it’s described in the book is a heavy focus on pure enablement. Now this is the one I might disagree with a bit in terms of where I see the value. I’d view this team as one that can connect business problems with data solutions. They can span the entire stack.

Defining Your Data Team’s Role and Vision

These categories don’t capture every possible type of data team, but they offer a useful starting point. They help data and business leaders clarify how different teams create value and what role each should play within the organization.

Once you’ve identified where your team fits, the next step is to define what success looks like for that role, both for the business and within the data organization.

Ask questions like:

• What outcomes is this team accountable for?

• How does it interact with other teams?

• What capabilities or resources does it need to deliver impact?

• How will we measure progress beyond technical metrics?

From there, you can start shaping a clear vision and the pillars your data organization will stand on. This should focus less on the motions or daily activities and more the outcomes you want your team to drive.

Making This Framework Actionable

Understanding your team type is only step one. Once you know it, you can start making decisions on where you can drive value.

Set Expectations With Leadership

I’ve seen many data teams get trapped as catch-alls. Automation, AI, data engineer, analytics, enabling, platform, etc.

In some cases, well, when you’re a team of one you’ll have to do it all. But there is only so much any team can get done and the more you try to increase that scope, the more likely you’ll start putting out bad work.

Or, on the flip side, you’ll succeed and then be stuck maintaining work that doesn’t drive much impact but two people say they really need.

So as you’re setting expectations, here are some tips:

• Don’t promise “strategic insights” if you’re really resourced as a platform team

• Frame success in terms of the value your team type is best positioned to provide

Prioritize the Right Work

Setting expectations makes it easier to prioritize the right work, because some work that comes your way will clearly not be best suited for your team. If your team is built up with DevOps and platform engineers, you won’t be as effective building front-end dashboards.

For example:

• Platform teams should focus on reliability and scalability, not ad-hoc requests

• Stream-aligned teams should bias toward speed and responsiveness

This is of course assuming your company is large enough to break down your data teams into these specialities.

Communicate Value in Business Terms

As someone who has worked on a platform team, it can be difficult to communicate your value. Often your work enables other teams to deliver value directly to the business. But you rarely get the credit unless you clearly tie it to what those teams are doing.

Here are some examples of how each type of data team might communicate their value:

• Enabling team: “We helped the sales org standardize KPIs, reducing weekly reporting time by 20 hours”

• Platform team: “We improved data freshness from 24 hours to near real-time, allowing the operations team to respond to issues”

• Complicated subsystem team: “Our demand forecast reduced stockouts by 15%, saving $5M.”

• Stream-aligned team: “Our campaign ROI dashboard helped reallocate budget, increasing efficiency by 12%.”

Closing Thought

In some organizations a data team is two people who spend half their time on analytics and the other half on IT workflows. While larger and more data-critical organizations have teams broken down into specific types.

Regardless, if you’re not careful all these teams can start to lose their purpose. In some cases they become catch-alls and in others they become so divorced from the business that they don’t even drive value any more.

That’s why it’s important to start with a general why. Why does your team exist and what is it that you do for the business.

As always, thanks for reading.

Upcoming Data Events

• Seattle Happy Hour At The Rhein Haus With BMC(I’ll Be There!)

• Airflow Summit 2025(I’ll be there!)

• Streaming Kafka Data into MotherDuck with Estuary Flow

Articles Worth Reading

There are thousands of new articles posted daily all over the web! I have spent a lot of time sifting through some of these articles as well as TechCrunch and companies tech blog and wanted to share some of my favorites!

The Knowledge Architect’s Playbook: The Pedantic Medallion

By

Let’s be honest with each other. Is our expertise a bridge, or is it a wall?

In our industries, we are paid for what we know. But we build value only when that knowledge is shared and understood. Too often, we build walls. We hide behind jargon, complex diagrams, and esoteric tools, not to create clarity, but to signal our own intelligence. We speak from a level we believe proves our worth, forgetting that the goal of communication is not to broadcast our own brilliance, but to build a shared reality with others.

Read More Here

The “D” In Data Stands For Discipline

A seasoned data leader once told me something that stuck: the d in data stands for discipline.

Now to be clear, pretty much every field requires discipline.

But we often focus on the exciting parts of data work in articles.

Not the tedious, repetitive, mundane, just doing the right thing type work.

Read More Here

End Of Day 196

Thanks for checking out our community. We put out 4-5 Newsletters a month discussing data, tech, and start-ups.

If you enjoyed it, consider liking, sharing and helping this newsletter grow.

Share

“The default state of data teams is failure.”

This was in the early 2020s, when many data teams felt like they had ballooned in size. Everyone wanted to be data-driven and cheap money made it easy.

Fast-forward to 2025 and the landscape looks very different. Companies are running leaner. Many have intentionally shrunk their data teams and, in some cases, lean more on external partners instead of adding headcount.

As a consultant, I’m often brought in when a previous team has disbanded or when leadership wants to turn around a struggling data environment. Across these engagements, I’ve seen recurring patterns, root causes that explain why some data stacks and teams fail to deliver, and what it really takes to bring them back on track.

This is based off of my talk at Big Data London.

Root Causes of Data Team Failure

There are many challenges data teams and leaders face. But it’s not just about picking the wrong technology. Many data leaders find themselves leading data teams with minimal coaching after being an IC. Others get pulled into being a catch all team where they have to manage not only the reporting but automated processes that might be better suited for a different team while also having to figure out how to lead AI initiatives for their companies.

All that said, here are some common causes of data teams failing.

Lack Of Ownership

Lack of ownership can show up in many forms. I liked the way it was described by in his article 6 Archetypes of Broken Ownership. In which he references that ownership depends on three elements: mandate, knowledge, and responsibility. If any one of these is missing, true ownership breaks down.

Consider a leader who has the power to make a decision (mandate) but lacks deep understanding (knowledge) and isn’t accountable for the consequences (responsibility). They make a call without consulting the team.

The result?

• The team delivers the project but morale sinks

• People stop raising concerns because they feel their input doesn’t matter

• Attrition follows as talented engineers look for healthier environments

Other ownership gaps signs:

• Key assets, tables, dashboards, critical scripts, become orphaned when the original owner leaves

• Technical leads are responsible for data pipelines but can’t influence upstream data quality

• Analysts are tasked with metric definitions but can’t enforce consistent usage across departments

The truth is, like many things, lack of ownership by itself doesn’t usually cause a data team to fail. Instead, it’s the build-up of multiple issues.

Over-engineering And Wrong Sizing

I view this as a spectrum. Where you can either over-engineer and be burdened by infrastructure that is too complex for the problems you are trying to solve and on the other side have infrastructure that doesn’t scale well forcing you to fight fires and deal with technical friction.

In both cases you often lose the ability to cycle quickly through iterations. Instead of experimenting, measuring, and improving, the team is stuck fixing infrastructure, whether that’s debugging an overbuilt DAG dependency maze or scrambling to scale a single-node warehouse that keeps running out of credits.

Common over-engineering signs

• Chasing the “cool tool” of the moment without a clear business requirement

• Layering every emerging pattern (micro-services, event buses, lakehouse-plus-warehouse) before proving the need

• Designing for hypothetical petabyte scale when current data volumes are modest

Common wrong-sizing(and messy prototyping) signs

• Under-estimating data growth and user demand, causing constant outages or painful migrations

• Skipping basic reliability features to move fast, which later slows everything down

• Leaning on brittle quick fixes, one giant ETL script, a single overworked cluster, until making even a small change takes weeks

Both extremes carry the same cost, slower cycle times. Engineers spend their time nursing the platform instead of delivering new insights or helping the business adapt. Making small changes takes forever, even when all you need to do is add a few columns to a table.

Metrics Theater

This is one particularly frustrating because it often appears that the data team has done everything right. You’ve built reliable dashboards and a rock-solid analytics platform, yet no one is making real decisions based on the data.

Leaders may even review the numbers weekly or monthly. When metrics are positive, they celebrate. When they’re negative, they shrug: nothing we could have done.

It’s all an act. Metrics theater allows leadership to look data-driven without actually being data-driven.

Typical signs:

• Everyone says “we’re data-driven,” but key calls still come from gut instinct

• Metrics are highlighted when favorable and quietly ignored when unfavorable

• Reports are created and placed into slide decks, but the business is still facing the same issues

The result is expensive infrastructure and beautiful dashboards that fail to change how the business operates. If that’s the case, then the business is really just paying for an expensive paper weight.

Outputs Without Outcomes

Plenty of companies out there invest millions into their data stack. They build dozens if not hundreds of tables and pipelines, and focus on developing more and more.

Success is measured by what the team ships, not by what the business achieves. Although this is often more implicit. It might not be said, but what gets rewarded is output.

Typical signs:

• Number of dashboards, tables, or lines of code lead to promotions

• Launching data products that are rarely, if ever, used

• Spending millions on migrations or new tooling with no measurable lift in revenue, efficiency, or customer experience

I’ve seen teams build 100+ dashboards and complete multiple costly migrations, only to discover that decision-makers barely glance at the outputs.

Both metrics theater and output-focused thinking miss the point. Its a lot of motion without really any true purpose.

Turning the Ship Around - Your Data Team Turnaround Playbook

But that’s enough talking about problems.

Let’s go through how you can start to turn your data team around.

Diagnose & Stabilize

Before actually really diving deep you’re going to want to pause and just assess what is going on. You need to understand what are the crucial workflows you can’t break because they will impact the business? And you’ll want to see if there are opportunities to simplify but you can only do that by understanding the bigger picture.

• Map the current stack and data flows - Document pipelines, source system, and dependency. There will always be some hidden cron job or Snowflake task that you didn’t expect that somehow manages the way your business makes money.

• Identify “must-keep-lights-on” processes - Isolate the handful of jobs that keep revenue reporting, billing, or critical dashboards alive. In many cases data teams might be responsible for business critical workflows, you can’t mess those up.

• Deliver quick wins

  • Fix broken daily jobs and remove dead code

  • Standardize naming for tables so everyone can find and use data more easily(specifically target the tables that are heavily used)

  • Set up simple alerts and monitoring to prevent recurring outages

These early actions buy credibility with leadership and create breathing room for deeper fixes.

Reset Vision & Priorities

Once you’ve got a good idea and handle of the current data infrastructure it’s time to reset. You need to be clear of what your teams priorities are and what the vision for your team is in the future.

What are the 2-3 pillars you believe your data team should focus on?

Do you view it as purely a reporting team? a team that will help integrate data into the product, etc.

Be clear about it, or you’ll always end up a catch-all IT/Data/Finance team hybrid.

• Reconnect data work to business outcomes - Identify top three to five metrics or use cases the company truly cares about, such as reducing churn, improving conversion, or speeding new product launches.

• Publicly prioritize top initiatives with clear owners - Create a short, visible roadmap so the entire company sees what matters and what will not be done.

• Get leadership buy-in on stopping low-value work - Make it explicit, ad-hoc requests and “nice to have” dashboards are paused until the core plan is met. Ok let’s be real, most of the ad-hoc work…you’ll always have to deliver some or people will think you’re not willing to help them. But be willing to pushback more or you will never get out of the quagmire.

Rebuild Foundations

Now you need to rebuild the foundation. And I don’t just mean your technical foundation, I mean everything from technical to how you manage ownership, key processes, etc

• Establish clear ownership and accountability - Being clear on who and how ownership will work moving forward allows your team to make better decisions because they’ll know who is responsible for managing various future artifacts

• Simplify architecture and right-size tools - One example I have where I simplified architecture involved 50+ queries supporting a set of dashboards. I look through them and found they could be consolidated to 3-4 instead. Reducing duplicative code, improving performance and not hitting the data warehouse as often.

• Document, automate testing, and enforce standards - This is also a great time to set clear standards on everything from how you test to name tables and columns.

Create an Outcome-Obsessed Culture

Finally, I’ve really enjoyed the term outcome-obsessed recently. Probably because the alliteration. But the point remains, you’ve got to actually care about what the other side of your project does.

Where the data work stops the business value is just beginning and it doesn’t just happen because dashboard exists or a business executive has access to a chatbot.

• Define success in business terms, not just technical metrics - Yes, data quality has to be reliable, the dashboards need to load in a reasonable time frame, but the business still has to have a plan for the data.

• Bake outcomes into processes - If you’re only really starting to think about outcomes after delivering your data products, there is a good change you’ve already gone down the right path.

• Keep communication constant and keep the project going - Just because the data product is delivered doesn’t mean you don’t have to communicate with anyone in the business anymore. You need to check in, see if they are using the dashboard, see if they have any issues, and so on.

Final Thoughts

Data teams can provide a lot of value for the business but they can also very easily be an expensive paper weight.

They can find themselves building data pipelines that support dashboards that no one uses and arguing about whether or not they should migrate to Databricks or Snowflake next.

All while the business is talking about a completely different set of problems and issues that you could be helping on!

As always, thanks as always for reading!

Upcoming Data Events

• Streaming Kafka Data into MotherDuck with Estuary Flow

• Seattle Happy Hour At The Rhein Haus With BMC

• Airflow Summit 2025(I’ll be there!)

Articles Worth Reading

There are thousands of new articles posted daily all over the web! I have spent a lot of time sifting through some of these articles as well as TechCrunch and companies tech blog and wanted to share some of my favorites!

Operating Principles That Guided Me to Staff Engineer (Part 2: Expanding Influence)

By

In Part 1, Jordan shared the principles that helped him consistently deliver results and reach Staff Engineer. But results don’t happen in a vacuum. They depend on the influence you have to make change happen. Influence drives results, and results compound to more influence.

In this part, Jordan will share the guiding principles that helped me build influence to make a multiplicative impact across teams. These principles help you build relationships, strengthen your communication, and be seen as a leader.

Read More Here

Scaling Muse: How Netflix Powers Data-Driven Creative Insights at Trillion-Row Scale

At Netflix, we prioritize getting timely data and insights into the hands of the people who can act on them. One of our key internal applications for this purpose is Muse. Muse’s ultimate goal is to help Netflix members discover content they’ll love by ensuring our promotional media is as effective and authentic as possible. It achieves this by equipping creative strategists and launch managers with data-driven insights showing which artwork or video clips resonate best with global or regional audiences and flagging outliers such as potentially misleading (clickbait-y) assets. These kinds of applications fall under Online Analytical Processing (OLAP), a category of systems designed for complex querying and data exploration. However, enabling Muse to support new, more advanced filtering and grouping capabilities while maintaining high performance and data accuracy has been a challenge. Previous posts have touched on artwork personalization and our impressions architecture. In this post, we’ll discuss some steps we’ve taken to evolve the Muse data serving layer to enable new capabilities while maintaining high performance and data accuracy.

Read More Here

End Of Day 195

Thanks for checking out our community. We put out 4-5 Newsletters a month discussing data, tech, and start-ups.

If you enjoyed it, consider liking, sharing and helping this newsletter grow!

Share

Before diving in to that, I wanted to let y’all know that I’ll be running several events in person in the UK, Seattle and Denver. The first will be in the UK, so if you’d like to join me after Big Data London, you can sign up here!

Also, special thanks to Estuary for partnering on this event!

Food and drinks will be provided as well as a live band!

Now let’s jump into the article!

A seasoned data leader once told me something that stuck: the d in data stands for discipline.

Now to be clear, pretty much every field requires discipline.

But we often focus on the exciting parts of data work in articles.

Not the tedious, repetitive, mundane, just doing the right thing type work.

It is those unglamorous but critical habits that help you deliver good work:

• Resisting the urge to over-engineer a simple problem

• Enforcing standards even when it feels tedious

• Saying “no” to the ad-hoc requests that would derail your team from finishing meaningful work

• Committing to an idea even if it doesn’t provide immediate results

None of these are flashy. They don’t exactly make for a good case study or talk at a conference. But looking back into my article from a couple weeks ago. But they are more habits that I’ve seen separate good and great data teams.

There are plenty of benefits from doing all the small things well.

Below are three ways discipline shows up in a high-performing data teams.

Practicing Restraint In Data Infrastructure Choices

Here I often think back to my early cooking days. As a young cook, I’d pile every technique I knew onto a single plate. I wasn’t thinking about the diner’s experience. I really was just showing off.

Over time I learned that every component must earn its place.

In the same way, data infrastructure is a tempting place to use tool after tool and build a massively complex system to report churn.

It’s tempting to keep adding tools, Iceberg, Databricks, Snowflake, Airflow, Sigma, Unity Catalog, until the stack looks impressive on paper. Yet, I’ve seen teams spend millions building these sprawling setups only to hear the same complaint from business leaders:

“I can’t find the numbers I need”

So technically, you might have built an amazing data infrastructure stack but functionally for the business no one uses it.

So why did you build it?

Saying No to the Endless Ad-Hoc Ask - And Actually Finishing Work

Data teams can quickly become the company’s catch-all IT fill-in department.

One day it’s a finance team automation, the next it’s a one-off dashboard or an urgent “just pull this number” request. Each ask might seem harmless, but together they create a flood that pulls the team away from meaningful, long-term work.

Data teams need to have the discipline to say “No”.

Now I say “No” but I mean, you need to be able to communicate what your team can and can’t get done and what trade offs are.

It’s also about ensuring your team isn’t constantly getting ripped from one project to another.

After all:

“you only get value from projects when they finish: to make progress, above all else, you must ensure that some of your projects finish.”

― Will Larson, An Elegant Puzzle: Systems of Engineering Management

If you let your data team get steam-rolled and just say yes to every ad-hoc request, at first other leaders might enjoy how much attention they are getting, then they will keep asking for more and more and your data team will get burnt out, projects won’t get fully delivered and things will just continue to unravel.

Following Standards Relentlessly

Standards cover everything from how tables are named to how version control is handled, how SQL is formatted, and how data models are structured.

I don’t view these are not just cosmetic choices. Consistency ensures that a query written today will still be understandable (and trustworthy) six months from now.

Now in the past this meant I’d have to go over my SQL scripts and ensure that I tabbed everything as expected.

Luckily now you can automate this part of “discipline”. You can do so by making those standards part of the process. Use linting tools, CI/CD checks, and code reviews so that every pull request is validated before it ever hits production. When we did this at Facebook nits went down 98%(Or at least that’s what it felt like).

Documentation is part of the standard. A living style guide and onboarding guide help new team members ramp up quickly, reducing the “legacy knowledge” that can paralyze a team when someone leaves.

This also makes it easier for external team members to quickly understand what your naming conventions and workflows look like.

It also makes future migrations, looking through code for specific patterns and a whole host of other larger projects considerably easier.

But Wait There Is More!

Now there are plenty of other ways data teams remained disciplined when it comes to delivering the right end products to the business. If you’d like to read some of my past articles on some of the topics you can see them below.

• Keeping focused on the business

• Not letting hype derail what actually matters

• Building production ready data pipelines

Final Thoughts

So many projects and data stacks I’ve seen fall apart because of small compromises that built up over time.

• There was no clear owner for a task or larger process.

• The data engineering team used a tool to patch a problem rather than inquiring why they were having the issue.

• No one set standards which allowed for fast development at first until the first migration.

Each small compromise seems harmless in the moment, but together they create technical debt, brittle pipelines, and future pains.

There is always a balance of course, you don’t want to put so much process in the way that nothing moves.

But you also want to avoid building a big ball of mud that no one wants to touch.

As always, thanks for reading.

Upcoming Data Events

• Airflow Summit 2025(I’ll be there!)

• London Is Calling - Data And Drinks With Estuary

• Coffee With The Seattle Data Guy - London

• Big Data London(I’ll be there!)

Articles Worth Reading

There are thousands of new articles posted daily all over the web! I have spent a lot of time sifting through some of these articles as well as TechCrunch and companies tech blog and wanted to share some of my favorites!

Is TV's Golden Age (Officially) Over? A Statistical Analysis

By

A lot can change in forty-five months. Think back to November of 2021: the world had yet to see a Tesla Cybertruck, HBO Max was an ascendant streaming service, Will Smith had slapped zero people on live television, Sam Bankman-Fried was a benevolent billionaire and model citizen, and Netflix's stock was soaring, buoyed by the pandemic.

And then the unthinkable happened: Netflix reported a quarterly loss of 200,000 subscribers. This earnings miss—coupled with broader economic uncertainty—triggered widespread panic across the entertainment industry. Streaming platforms slashed their content budgets, media conglomerates like Disney and Warner Bros. laid off thousands, and Netflix's stock fell by 51%. This industry-wide contraction culminated in a six-month writers’ strike, as unions demanded higher pay, standardized compensation, and greater residuals. Amid full work stoppages and a volatile economy, industry pundits began speculating whether streaming could recapture its pre-2022 momentum.

Read More Here

The Pedantic Layer

By

In the data world, we love to argue about definitions. What is unstructured data? Is JSON structured or semi-structured? Are PDFs unstructured, or do they contain “implicit structure”? Do LLM embeddings of text count as structured data? And WTF is a semantic layer? Entire threads, articles, and even conference talks spin out of these debates.

This obsession forms what I call the Pedantic Layer…

Read More Here

End Of Day 194

Thanks for checking out our community. We put out 4-5 Newsletters a month discussing data, tech, and start-ups.

If you enjoyed it, consider liking, sharing and helping this newsletter grow!

Share