Snowflake vs Databricks Is the Wrong Debate
Winning the Data Stack Role by Role
Over the last few years, Databricks has been executing a strategy to take over the entire data workflow.
Maybe it never started that way.
Maybe when they first came out, they only ever planned to be a managed Spark solution. But I have a hard time believing that, mostly because I believe their leadership has the vision and capabilities to see far beyond that.
Databricks has always been pretty upfront that they want to be the end-to-end data stack. But they’ve been approaching it piece by piece.
Or should I say role by role?
Obviously, at first, their focus was on the data scientist and ML engineer.
But in 2020, they wanted to shift the narrative. They were more than just managed Spark, they were a data platform that could replace your others. So they championed the idea of the Data Lakehouse. You could view this as a capturing of the data engineering market.
But this didn’t happen overnight.
If you look at some of the posts and content shown below, Databricks tends to push an idea or concept very hard when they really want to capture that market. Shown below when they pushed the concept of the Data Lakehouse and now their data visualization and analytical workflows.
I’ve already written about the Data Lakehouse and how it was heavily pushed by Databricks in 2020, so I don’t want to dive too deeply here. I believe there are far more interesting recent happenings that stick out.
This past month, I believe Databricks started to make its final push into gaining the mental market share for analysts.
How?
Databricks just partnered with the biggest analytics creator.
Alex The Analyst.
There are several reasons I believe this is significant.
Although many companies partner with data creators all the time, I think, if we look at Alex the Analyst’s audience, we’ll see that Databricks wants to put the final nail in the coffin of the data analytics workflow. They want to show end-users that hey, if you’re an analyst, Databricks is for you too! We aren’t just for data engineers; we can help you drive value, and we aren’t too complex.
Databricks did something similar when they wanted to win over the Data Engineering space. They partnered with Bill Inmon to put out a seminal piece on the Data Lakehouse. This was about a year after Databricks wrote their original piece, and it wasn’t until this piece came out that they finally got real traction.
Databricks has been pushing its data visualization solution for a long time now. But now, they want to get even better traction by getting the face of learning analytics out there.
I wouldn’t be surprised if one reason they are doing this is that they are likely continuing to get pushback from customers who are more of the analyst type, who believe Databricks isn’t for them. In fact, I have worked on several projects over the past year alone where that was the sole reason for a Databricks and a Snowflake environment.
In these cases:
Snowflake was for the analysts.
Databricks was for the engineers.
I think the other point that stands out is how heavily the Databricks visualization tool was shown in Alex’s content. Now, part of this could have been because that’s what Alex felt his audience would appreciate.
But here is another anecdote I’d like to share.
I’ve spoken to several data leaders who have told me that Databricks account executives tried to push them very hard on their visualization tool. In some cases, to replace their Tableau or Power BI instance.
I think account executive interactions tell you a lot about a company’s strategy. They are the front line; what they are doing is literally a reflection of where the company is trying to go. If the company wants to push a new feature, a new partnership, or a new product, you push it through the account executives.
What Does This Mean For The Data World?
Ok great Ben, why does anyone of this even matter?
Here are a few thoughts on where this is all going and why it’s important.
Snowflake Vs Databricks Is A Red Herring
The real vs. conversation isn’t Snowflake vs Databricks, it’s really Databricks vs Microsoft, AWS, and even Salesforce.
Let me explain.
These are more generic clouds and SaaS, but they also offer either components or the entire data stack.
More importantly, they are often solutions you pick prior to thinking about your data tooling. You likely use Azure to set up your application or Salesforce as your CRM. So, when it comes time for data analytics, you already have Azure, so just use Power BI; it’s already part of your contract anyway (I’ve had multiple data leaders give me that line of thinking when it comes to picking BI).
Why not use Tableau? You’re already using Salesforce. Will it probably make negotiations better, right?
Of course, there is still some Snowflake vs Databricks. But for those 8 and 9-figure deals, you’re fighting much larger companies. Companies that could buy you out.
But this leads to my next point.
Databricks Is Building The Next SAP
I referenced this idea at the bottom of a post on the Data Leaders Playbook that Snowflake and Databricks are building SAP backwards. Instead of going from business applications and eventually building out solutions like SAP HANA.
They are going from data analytics to the business.
With the recent purchase of Neon, Databricks can now enter the sales conversation earlier, which I think is arguably the real point. Yes, yes, all the tech people are pushing back their glasses and about to say something pedantic.
If you want to win the CIO over, and not just the data team, you have to enter the conversation earlier. Meaning the application layer. I am sure some Palantir fans in the back are just waiting to start writing a comment(This has been Palantir’s goal the whole time!).
Let’s put that aside for now and just think about the company these businesses are selling into. Most companies, especially digital ones, have a few key databases. The ones that actually represent their main application or service. It’s where most of their data lives. If Databricks can get you to use their Postgres instance, then they already have you in the funnel to get you to use their data Lakehouse, then they already have you in the funnel to use their BI, their data pipelines, etc.
So I expect they will be pushing this space hard in the next year or so. I’ve actually already seen this hinted at via some of their distribution and creators they enjoy working with.
If You Want To Win Mental Market Share You Have To Be Relentless
Databricks was pushing the Data Lakehouse concept for over a year until it really started to gain traction. I imagine it required millions in terms of employee time, partnering with consultants, etc.
You can’t just create a term or want to gain entrance into a space without having a plan on how to win.
I say this because I’ve spoken with so many marketing teams and leaders who have wanted me to do “un-boxings” of their product as a one-off.
That’s a tactic, not a strategy(and not a very well thought out one at that).
One video doesn’t move the needle. Think about how many posts, videos, talks, consulting partners, and so on, Databricks has been pushing to talk about their AI/BI Genie and Data Visualization.
Just paying for distribution on a one-off piece of content won’t work.
Even now, with the recent partnership with Alex. This doesn’t feel like a one-off. Databricks keeps promoting the content. On their LinkedIn page, in their product.
They’ve got the employees posting about it. They are trying to make Fetch happen, and they know this isn’t going to be easy.
They want analysts to come to Databricks, and when they do, they want them to see a familiar face.
If done well, it can pay off. I doubt you can fully attribute Bill Inmon to shifting 100% of the conversation to the Data Lakehouses, but if he even had a 5-10% impact, that’s a multi-million dollar impact.
Databricks wants to do that again with the analyst.
Final Thoughts
Something tells me that Databricks will be pushing Alex’s videos hard for the next few months.
It’ll complete the trio of data roles.
In turn, this allows Databricks to start to focus on the application layer.
To start having conversations earlier with business, not just about data strategy, but about IT and business strategy.
As always, thanks for reading.
Video Of The Week - 5 Things in Data Engineering That Still Hold True After 10 Years
Articles Worth Reading
There are thousands of new articles posted daily all over the web! I have spent a lot of time sifting through some of these articles as well as TechCrunch and companies tech blog and wanted to share some of my favorites!
Is It Time to Say Goodbye to Data Engineers?
Ever since tools like SSIS came onto the scene, vendors and business leaders have been on a mission to remove what they see as the biggest roadblock to data-driven decision-making: data engineers.
Or their counterparts—DBAs, ETL Developers, and Data Architects.
Sure, not everyone says it so explicitly, but you can see it in vendor marketing and in the decisions made by the business.
I remember talking to a veteran data expert who’s been in the field for three decades. They told me that when SSIS first launched, people were genuinely afraid for their jobs. The idea that you could just drag-and-drop tasks that once required code was nerve-racking. But if you’ve used SSIS, well, you know the truth.
To some extent, I get why the idea is appealing. When a leader requests a report, a software engineer wants to modify an application table, or a data scientist wants to explore a new dataset, who’s the one slowing down the project?
The data engineers.
Refresher on Experimentation
As has become a tradition, I end the year with a refresher: one single, consolidated guide that brings together the most important definitions, reporting best practices, example dashboards, and key principles - all in one place.
Last year, I published these three:
This week, I’m sharing a refresher on A/B Testing. I believe I published one before, and I also have a dedicated experimentation section in my newsletter. This time, I expanded it with more resources to help you properly get started with A/B testing. It covers tools, core concepts, free classes, and more.
End Of Day 205
Thanks for checking out our community. We put out 4-5 Newsletters a month discussing data, tech, and start-ups.
If you enjoyed it, consider liking, sharing and helping this newsletter grow.
Once the Databricks bill hits a certain threshold, it’s no surprise to hear executives ask: 'Why are we still paying for Power BI licenses (and premium capa) if we already have these capabilities in Databricks? (propably with a gently push from databricks sales team :) )