Can Palantir Take Over The Entire Data Stack
Or Is It Too Late To The Modern Data Stack Party
This week I have been digging into Palantir’s tool called Foundry. I have buried myself deep into Stack Overflow questions about Foundry, looked into their AWS integrations, and watched multiple CGI and product demos of how companies could use Foundry as an all-in-one data tool.
Why?
Because I realized that Palantir is trying to take on the entire modern data stack. All in one tool.
Don’t believe me? Just check out some of their marketing materials on the Foundry main page.
They think they have the answer to analytics and the entire data infrastructure stack.
I guess I should stop writing my series on the baseline data stack huh?
But let’s drill into this more before I give up on the last few articles.
Let’s start with a broad overview by connecting Palantir to something we all know.
AWS.
🔌 Looking Into How Palantir Connects To AWS
Let’s go through Palantir’s Foundry diagram they have on AWS. Looking below we can see that Palantir’s Foundry attempts to take on the entire data stack. Starting with ingesting data into Foundry they also have transformed and then develop an ontology (which I assume is supposed to act as a generalized data model).
Essentially they are taking on Fivetran, Informatica, and every other data pipelining tool while trying to take on Snowflake and dbt.
From there they even seem to have some components that deal with MLOps concepts like monitoring and model deployment.
Finally, they also have an application layer that based on their video demos looks like it could be similar to Retool. Based on their videos it looks like on the application side they might be providing an easy drag and drop application development platform.
Overall, it looks like they are taking care of the whole nine yards. Everything. Throw away your current duct-taped together data stack.
And just use Foundry.
🤔 Palantir Has Several Issues Though And It’s Not Tech
If it wasn’t for WallStreetBets, would people even really know what Palantir does?
Ok, arguably they have had plenty of traction thanks to their connection with Peter Thiel and all of their military and government contracts.
Messaging And Marketing
Despite this, I do feel like Foundry has gone a bit under the radar. Perhaps they have been doing too much to try to sell all of their value. Shoving a 10-year marketing plan into 1.
In this regard, Palantir reminds me of DC chasing Marvel’s cinematic universe.
Maybe their goal isn’t to be as ubiquitous. Where tools like Snowflake have hired droves of developer advocates to create content as well as not limiting their product to billion-dollar companies.
Palantir puts out the occasional video or Medium article of very specific use cases that can be awe-inspiring but also very niche. And of course, constantly seeing Alex Karp on CNBC.
Looping back to the billion-dollar company reference. It does seem like Palantir hasn’t focused on smaller companies. They do tend to have a specific client size.
🧑🤝🧑 Limited Sets Of Clients At The Right Price Point
Based on Palantir’s average contract size it seems as though their main focus is on larger companies. For example their average contract size is at least 5-6 million.
This might be limiting the ubiquitous feeling other tools have.
That being said, Palantir has attempted to involve itself more heavily with smaller companies. For example, last year Palantir launched Foundry for Builders, an initiative dedicated to supporting early-stage companies by providing them with the Palantir Foundry platform
The first few groups of companies are start-ups in varied sectors, ranging from healthcare to robotics, to software and fintech. They are:
Chapter, a New York-based company that helps customers understand and select Medicare plans
Hence AI, based in London, uses AI to help companies make better relationship decisions in the legal and consulting industries
Adyton, a Scottsdale-based builder of mobile software that links users in the field with enterprise systems
A fintech company based in Oslo, Norway, that aims to simplify B2B transactions by offering a buy now, pay later alternative
Gecko Robotics, a Pittsburgh-based maker of robots used for industrial inspections
Considering this program started back in mid-2021 it does make sense that Palantir is still getting its feet wet in the smaller contract space. So maybe we will start hearing more about Palantir this year.
🤝 Lack Of Partnership Networks
Another problem I noticed is Palantir doesn’t seem to like to play the partnership game as much as other technologies. GCP has a highly developed partner program that helps push their products through solutions consultants.
Going on the Palantir’s site the partnerships that I did find were related to military or government intelligence and there doesn’t seem to be as many commercial arms.
That being said, if you look at the picture below you will see that their partner SHINE Systems does provide Palantir training. As does Palantir itself. Alluding to the fact that their is interest in expanding the user base that can build on top of Palantir.
However, the drop-down menu on the page for Palantir training suggests it is for customers and for people such as myself who would like to learn more about their tool.
At the end of the day, this might be ok. Perhaps Palantir will go the Epic Systems route. Where the only people that can work on Epic Systems are people who were directly certified by Epic. Epic Systems is a very tightly managed certification process. But with its near monopoly-like status in the medical world, Epic gets to set the rules.
Now there is one partnership they did take on last year that stuck out. They partnered with DataRobot which is one reason I was slightly surprised when I realized Palantir was trying to do more than just AI and machine learning.
The goal of this partnership is to take the models from DataRobot and bring them back into Foundry. Infusing them into operational workflows, delivering massive scale data and AI to business users. The models are constantly updated and trained by DataRobot to keep them relevant and fed back into the organization’s integrated data asset. - DataRobot
So maybe Palantir can play nice with other companies.
⛏️ Let’s See What Else We Can Dig Up
To be honest, I was limited on how deep this initial pass went into Palantir and where it fits into the data world. I have never had the chance to work with Palantir in any form so I did most of my research via documentation, press releases, Stack Overflow questions, and so on. I would love to learn more.
Can Palantir compete with Snowflake and Databricks as Alex Karp discussed in a recent interview? Are they even really even trying to solve the same problems?
And even if they are, is Palantir too niched to compete?
I want to point out that I am looking at Palantir from a very holistic standpoint. I am not just interested in their technology but also how they are trying to position and take market share.
If you have any perspectives on Palantir or have worked with Foundry I would love to hear from you. Feel free to jump on my Discord and reach out to me!
Do You Love Your Data Stack?
That might sound like a weird question, but Rivery interviewed dozens of data team members at all sorts of companies, and heard a common theme; “There’s a lot of pressure to deliver results quickly, and then management wonders why stuff breaks and changes take so long.”
It turns out that lots of people don’t love their data stack, nor do they enjoy the process of piecing together separate tools, while trying to debug issues and crank out ad-hoc reports.
Rivery’s fully-managed SaaS ELT & DataOps platform just might be the data stack you’re ready to “settle down with,” and right now is better than ever to try it out.
Sign up to create a free Rivery account and get $1,500 worth of Rivery usage credits and a complimentary session with a Rivery Solutions Engineer to help you with your specific needs. This offer is for new Rivery users. Promotion ends Feb. 24th, 2022.
Sponsorship
Special thanks to Rivery. Rivery’s SaaS platform provides a fully-managed solution for data ingestion, data transformation, data orchestration, reverse ETL and more, with built-in support for your data operations development and deployment lifecycles.
Video Of The Week: SQL Interview Questions For Data Scientists And Data Engineers
Join My Data Engineering And Data Science Discord
Recently my Youtube channel went from 1.8k to 21k and my email newsletter has grown from 2k to well over 5.5k.
Hopefully we can see even more growth this year. But, until then, I have finally put together a discord server. Currently, this is mostly a soft opening.
I want to see what people end up using this server for. Based on how it is used will in turn play a role in what channels, categories and support are created in the future.
Articles Worth Reading
There are 20,000 new articles posted on Medium daily and that’s just Medium! I have spent a lot of time sifting through some of these articles as well as TechCrunch and companies tech blog and wanted to share some of my favorites!
3 Innovations While Unifying Pinterest’s Key-Value Storage
Engineers hate migrations. What do engineers hate more than migrations? Data migrations. Especially critical, terabyte-scale, online serving migrations which, if done badly, could bring down the site, enrage customers, or cripple hundreds of critical internal services.
So why did the Key-Value Systems Team at Pinterest embark on a two-year realtime migration of all our online key-value serving data to a single unified storage system? Because the cost of not migrating was too high. In 2019, Pinterest had four separate key-value systems owned by different teams with different APIs and featuresets. This resulted in duplicated development effort, high operational overhead and incident counts, and confusion among engineering customers.
Reviewing Varada And How It Can Improve Trino’s Performance
Companies are continually setting up programs and initiatives to become more data-driven. Millions are being spent on new infrastructure, hiring employees, and creating processes to drive value with data.
The need for better tooling, faster data, and experienced data professionals will become even more important as data sources, size, and speed grow. However, eventually, the amount of data engineers you need to hire becomes unsustainable. With the average salary in the US for a data engineer around 90-150k, companies may struggle to manage large data teams.
Thus, in the future, companies will likely be forced to rely on better tooling to help reduce the amount of manual intervention and there seems to be a growing number of them.
Project RADAR: Intelligent Early Fraud Detection System with Humans in the Loop
Sergey Zelvenskiy, Garvit Harisinghani, Tiffany Yu, Edwin Ng, and Robin Wei
February 1, 2022 0
Uber is a worldwide marketplace of services, processing thousands of monetary transactions every second. As a marketplace, Uber takes on all of the risks associated with payment processing. Uber partners who use the marketplace to provide services are paid for their work even if Uber was unable to collect the payment. Fraud response is thus a very important operational component of Uber’s global marketplace.
Industry-wide, payment fraud losses are measured in terms of the fraction of gross amounts processed. Though only a small fraction of gross bookings, these losses impact profits significantly. Furthermore, if a fraudulent activity is not discovered and mitigated immediately, it could soon be further exploited, resulting in serious losses for the company. These dynamics make early fraud detection vital to the company’s financial health.
End Of Day 35
Thanks for checking out our community. We put out 3-4 Newsletters a week discussing data, tech, and start-ups.
If you want to learn more, then sign up today. Feel free to sign up for no cost to keep getting these newsletters.