Hi, fellow future and current Data Leaders; Ben here 👋
Today I wanted to discuss spreadsheets and well Excel. It’s a tool that continues to be used decades later by both non-data and data teams alike. For some, this is a frustrating fact as they think everything should be in code and version control. For others they are making it part of their workflows.
But before diving into today’s newsletter, I want to take a moment to thank this issue’s sponsor: Hex. Hex brings the magic of AI to data analysis workflows, whether you’re using code or no-code. Hex helps organizations work together with data and avoid jumping between different data tools for querying, data science, visualization, and spreadsheets. Over 1600 organizations use Hex to do everything from deep analysis to self-serve.
Now let’s jump into the article!
Can we get an Excel export?
For many data teams, this ask can be somewhat frustrating. You’ve likely poured hours or days into a dashboard only to feel as if the team that asked for it just wanted an Excel report.
Then, of course, there are all the one-off actual Excel reports and lookup tables being managed by a single person in companies. In fact, at many companies, it’s these one-off Excel reports, VBA scripts, and Google sheets that run whole departments from operations to accounting.
Not your polished, QAed, dynamic and impactful dashboard!
It really can feel like the meme below.
It might make many new and more experienced data professionals scratch their heads. We have all this technology, data platforms that can crunch more data than ever before, and yet, somehow, everything seems to continue to touch Excel.
I recently had one person show me the financial model they built in Excel using Claude. So, like the QWERTY keyboard, spreadsheets will be with us if we ever manage to set up a colony on Mars.
And like the QWERTY keyboard, there will always be a minority resistance group that complains about its inefficiencies.
But Excel and spreadsheets, for many companies, is the way they are data-driven. It is the way their execs make decisions.
With that, I wanted to discuss some of the realities of Excel and spreadsheets and where I see them going in the future.
Excel Is A Blackhole
It’s tempting to fight Excel after all, if everyone is just slicing and dicing data, it’s very easy to start getting inconsistent metrics, out of sync data sets, and just create confusion.
Even if you built the perfect data analytics platform with every data point checked, once it’s out Excel who knows what will happen to it.
But it’s where a lot of “self-service analytics” actually happens.
And here are a few reasons why:
There Is A Low Floor And High Ceiling - Meaning, if you’re just getting started, you can use the SUM() function and slowly work up to VLOOKUPs, and for some people, you can take it to the point of building a rollercoaster simulator inside it.
It’s Far Easier to Edit - If all I want to do is add $50,000 to a single field, it’s fast, easy, and doesn’t impact the permanent data source like you would if you were to try to do the same in your data warehouse.
Instant What-If Analysis - Most BI tools aren’t really built for exploratory analysis. Excel? You can copy a tab, tweak three numbers, and immediately see the new forecast.
“Hey, what if our sales go up 12%… no, wait, 15%… actually, what if churn drops by 2%?”
No Gatekeepers - In many large enterprises, if you go ask for data, ask for a dashboard, ask for new columns to be added to a data set. You get push back. So instead, you go to Excel, pull the data sets that you do have access to, add your own columns and build your own solution. Now you can get your job done without feeling like it’ll take six months.
Many of the reasons above are why both data analytics and governance teams often find Excel difficult to grapple with.
Who hasn’t had to deal with a one-off Excel lookup table that only a single person manages?
Do you put it into a database table so the data team can now manage it?
Do you want to be responsible every time there is a new field?
Maybe you’ll create a form or tool they can edit to manage the data in the table themselves(although that can be quite heavy).
And what about the other 100 Excel sheets that are doing something similar?
I suffered the same issue at Facebook, and for a while, we actually did have a tool that allowed me to build a pretty simple table app that allowed an end-user to essentially manage an isolated database table directly.
Which was great, I could control columns, data types, and even add in a few rules to check what data was being inserted while the end-user was working in a spreadsheet-like interface.
But eventually that got deprecated, and suddenly I was once again left with other teams’ Google sheets(this was after we started reducing our use of Quip).
So, the spreadsheets once again reclaimed their data.
Maybe Excel Is a Symptom of Deeper Issues
In many ways, I think it can be argued that if everyone is trying to circumvent your dashboards and go straight to Excel, that might be the signal for a larger issue. Here are just a few reasons they might be deciding to go straight to Excel:
Your dashboards aren’t answering the real questions
They need flexibility that dashboards don’t offer
They don’t trust your source of truth(this would be a major issue)
They can’t wait two sprints for a dashboard tweak
They just want to build things themselves
The dashboard is slow
They feel safer making mistakes in Excel than in a tool they don’t fully understand
They don’t want to watch your video on how to use the dashboard(or feel like they don’t have time)
I’ve been in situations where the marketing, finance, and operations teams use multiple dashboards to extract a few key datasets and then put them into their own spreadsheets. That should be a signal that the dashboards you’ve built may not be meeting their needs.
I think it’s important to understand these issues so you can:
Build better dashboards that can answer your end-users problems
Guide partner teams to better short and long-term solutions(if you know that your partner team is just
Prioritize fixes that actually unblock how teams make decisions
Set expectations on what BI tools are for and what they are not
Know when to build a more flexible interface instead of another static report
Spot patterns in how different teams work and design solutions that fit their realities
And I am sure you could still list more reasons. The truth is, there is more than one reason a team might decide not to use your dashboard or data product. You should figure out why to see if there are specific issues with the dashboard itself, or if there are more uses that the team needs for the data.
If You Can’t Beat Excel, Include It
Recently, several BI tools have leaned into spreadsheets. Which I believe is a smart move for them in terms of increasing the user base and meeting people where they are.
As many BI tools often require some level of SQL which sure you could ask everyone at your company to learn, but I don’t think it’s fair to expect(not unless data teams also understand the business better).
So seeing companies provide a layer of familiarly in the BI layer makes some sense. Maybe it’ll finally get us closer to actually being able to put governance around spreadsheets rather than fighting them…
I’ve seen this massively increase the number of users who interact with BI solutions because the users can finally do what they want.
Write data back and manage ad-hoc lookup tables
Create formulas and calculations as they do in spreadsheets
Explore the data with the same freedom they’re used to, no rigid dashboard filters
Build quick “what-if” scenarios without submitting tickets or waiting for dev cycles
Annotate, comment, and explain numbers directly in the UI
I think most people would be surprised by what even a few small changes do in terms of making it easier to integrate workflows into your tool. Is it going to make data governance happy?
Probably not.
Attempts To Reduce The Chaos
Like I said earlier, we’ll likely never get rid of Excel or spreadsheets in general. It’s too convenient, too easy, solves immediate problems too well that if your goal as a data leader is to reduce the amount of spreadsheets that exist, its probably a fruitless effort. But I do think there are areas where analytics platforms can work to
Detect Ad-hoc Usage Patterns - This was one of the projects I was considering prior to leaving Facebook. I wanted to be able to detect how end-users were using data to see if there were ways we needed to extend the current data platform before end-users asked us. Maybe there are tables being joined or custom columns being created. The challenge with Excel and spreadsheets is much of this happens outside of any form of tracking. But I’d still love to see Snowflake or Databricks integrate this into their platforms.
Detect Metrics - This is one I’ve seen several tools and platforms offer. Where if they are seeing you define a metric, especially in a BI layer, they’ll suggest you create a universal metric and provide version control. I love the idea. After all, one of the challenges with some reporting and BI tools is they feel isolated. You create a metric in on dashboard and no other dashboard knows that it exists.
But none of this really brings Excel or spreadsheets under any form of governance or control. Not unless those spreadsheets exist inside some component of the data analytics platform itself.
Which is where I can see the world going. Especially if Snowflake and Databricks have their way and become the center for all your business data.
Excel - Villain, Symptom, or the Last Mile?
We often treat Excel like it’s the villain. Our nemesis in data.
In some cases, it can be. It can be a cause of many frustrating meetings where teams bring multiple conflicting metrics or maybe you offer to automate someone’s Excel process only to open up a 30 spreadsheet deep nightmare with more formulas than your VP has ‘quick asks.
But taking a step outside of the data team. For many, it’s their way of accessing data. Without anyone saying what they can and can’t do and you’re not going to take that away from them.
And, as the title suggests, many companies are run off Excel. So no matter how much some data teams want to put that genie back in it’s bottle, it’ll be around far longer than your dashboard.
As always, thanks for reading!
Events
Articles Worth Reading
There are thousands of new articles posted daily all over the web! I have spent a lot of time sifting through some of these articles as well as TechCrunch and companies tech blog and wanted to share some of my favorites!
5 Things in Data Engineering That Still Hold True After 10 Years
When I started in the data world back in 2015 Hadoop was at it’s peak.
Actually I happened to be scrolling through an old instagram account and found a picture from a DAMA conference where Horton Works sponsored it(you can see it below). At the time, Hadoop and its ecosystem were everywhere, Hortonworks, Cloudera, MapR, each promising to reshape the future of data.
Fast forward just ten years and many newer practitioners don’t even recognize those names. And yet, in data engineering, a decade is barely enough time for fundamentals to change. Underneath the hype cycles and new logos, many of the same challenges remain.
Cash Rules Everything Around Me
By Joe Reis
I’ve been reflecting on the last week’s announced merger of two darlings of the data industry, and observing reactions to this news. In the practitioner community, it’s definitely a mix of Kumbaya and dread. Plenty of people are hopeful things will stay positive, but nearly everyone is making a Plan B (forking, budgeting, looking at new options, etc). I’m agnostic to whatever happens, and try to take a Zen-like attitude to these things, mostly because stuff like this is very common in our industry. It is what it is, and here’s some advice if you’re new to the game.
End Of Day 203
Thanks for checking out our community. We put out 4-5 Newsletters a month discussing data, tech, and start-ups.
If you enjoyed it, consider liking, sharing and helping this newsletter grow.
A friend recently joined rowzero.com which attempts to solve some of the enterprise issues with Excel. Seems promising.
In B2B product management, marketing and sales, Excel is still the biggest competitor. Users know exactly how to get what they need out of it and how long it will take. A new entrant has to be orders of magnitude better to get someone to switch from what they know and and can count on.