The Problem With Buying Vs Building Your Software Services
What The Sales Executive Won't Tell You
CTO’s Focus of the Week
Have you heard of the ELTs, reverse ETLs, serverless databases, data lakehouses, or MLOps?
It’s always been hard to keep up in the tech industry when it comes to what the new best practice or solution is.
However, in the last few years, we have seen a proliferation of new technologies that can make it very difficult to keep up with the solutions marketplace.
Not to mention the recent marketing pushes to coin these various terms can be jarring.
This has made me think a lot about what are the right tools for which job.
More importantly, how do we as directors, developers, data engineers, and data scientists sift through all the noise of marketing and hype?
Let’s be honest, we have all been on a sales call where the account executive and sales engineer will convince you that their tool is the only tool for you.
They will show you prepackaged demos that don’t take into account the 999 ways the product could fail.
This isn’t to say that there aren’t a lot of great options when it comes to new tooling. Whether it be the modern data stack or the push for low-code services, there are always tools worth implementing in the sea of services and solutions.
In this article, we wanted to provide you some important considerations when buying your next service.
In addition, this is the start of a larger series. We are currently working on analyzing several categories of software services to help you, the reader, understand the current landscape of options. Currently, I am working on embed analytics solutions, breaking costs, features, and so on. Think Looker, Tableau, GoodData and Yellowfin.
What The Account Executives Don’t Tell You
Before signing any service agreement or contract. You need to consider why you are buying vs building. Again, account executives and sales engineers won’t tell you all the ways their tool can go wrong.
So here are a few struggles you will face with any service.
If Your Service Is A Small Or A Start-Up It Could Be Bought Out And Then Disappear
One problem with small start-up companies, in particular, is they can be bought out. This is a huge problem if you decide to use one of their services because the company that purchases them might choose to get rid of the company altogether.
Don’t believe me.
I was recently working on an article for a modern data stack solution that got bought out and will be shutting down their solution.
They had to inform all their customers that they would no longer provide them the service following a certain date.
Meaning that if you’re relying heavily on this product or service, you will have a very expensive migration project on your hands.
We are talking upwards of $50,000 migration costs.
So although you might think a start-up or small company may be able to give you a more personal experience.
There is a lot of risk.
Bugs Can’t Be Fixed By You - Isn’t That Good?
One of the selling points about low code solutions and the modern data stack is that your team won’t have to deal with fixing bugs.
At first, this sounds great and for the most part, it is.
If your team runs into a bug you can file a bug report and the service you are paying for will try to fix it quickly.
But what happens when they don’t?
Everyone assumes that services will likely fix bugs as soon as they are found out. However, from experience, I have worked with several services where they put off fixing bugs for months.
Although, understandably, teams are overwhelmed with new features to build and other bugs to fix this can become a major blocker to an organization that could cost them time and resources to either workaround or just wait for the fix.
All of which defeats many of the benefits that many of these automated modern data stack magic services discuss.
If you have a small development and engineering team, then this might not matter. If you would have built the solution in-house you would be dealing with the same problem.
However, if you have a well-developed engineering team that could have developed a more custom solution, then it might be better to consider building vs. buying.
Just so you can have control.
Using Low Code Solutions Is Always Less Customizable
Low-code/drag and drop tools boast the ability to automate workflows and reduce time to new features being developed.
From experience, these tools do make development faster.
Especially once you have gotten past the initial learning curve. Whether you are using a tool like Retool or Tableau all of these tools can help speed up the time it takes to develop new features and workflows.
They are also less flexible in terms of what they can do. You can’t go into Retool(A great low code solution) and reprogram some underlying feature as it’s not your code to alter.
This has always been a major issue when it comes to purchasing a software solution. In addition, the more the software solution tries to provide flexibility, the more developers will have to spend time learning the solution as well as configuring how it operates.
I will say that a lot of modern low-code options do provide more flexibility than some of their predecessors. I have used a few tools recently that allow developers the ability to connect API end-points either in terms of a CRUD operation or a data scrape for an ELT.
Even so, there are limitations and especially if you’re a developer or engineer by trade it can be very frustrating when you can’t optimize or improve service because you can’t access the code.
Overall, this is an important decision point when you’re trying to figure out what service works best for you(We are working on a guide to help you make better decisions in terms of the build or buy. Let us know if you’re interested).
Picking the right service isn’t easy.
You need to spend time thinking through why you are buying vs building but also what could go wrong in terms of the service disappearing or lacking the features you need.
Don’t just assume that the sales team knows what you need.
Their focus is generally to make the sale. Of course, the sales engineer will do their best to try to understand your use case, but there is always the motivation to close.
If you’re wondering whether your company is picking the right tool, then consider signing up for some office hours below!
Also, if you would be interested in a guide to help walk you through picking the right service or solution, then comment below just with a simple “yes”.
Ask A Data Consultant - Office Hours
Every newsletter I open up a day or two with a few slots for open office hours where my readers can sign up and you can ask me questions. I got to answer a lot of great questions so far and hopefully, they helped provide a lot of insights for those who signed up.
Sign Up Below:
Next Open Office Hours
For 2 Cups Of Coffee, You Could Help Support This Newsletter
If you’re interested in supporting this newsletter, then consider subscribing for a month!
Videos From Our Channel - Being A Data Engineer: Expectations vs Reality
Also, I don’t just put out a newsletter! I also have a Youtube channel that is slowly starting to take off.
A few weeks ago I put out a video about what it’s really like to be a data engineer.
Here is the description:
What is a data engineer?
The goals of a data engineer are much more big-picture and development focused. Data engineers build automated systems and model data structures to allow data to be efficiently processed.
This means the goal of a data engineer is to create and develop tables and data pipelines to support analytical dashboards and other data customers (like data scientists, analysts, and other engineers).
It's similar to most engineers. There is a lot of design, assumptions, limitations, and development that occurs to be able to create some sort of final robust system.
Articles Worth Reading
There are 20,000 new articles posted on Medium daily and that’s just Medium! I have spent a lot of time sifting through some of these articles as well as TechCrunch and wanted to share some of my favorites!
Solutions Architect Tips — The 5 Key Factors That Drive Application Cost
If you’ve recently made the move from developer to solutions architect, congratulations! Welcome to the technical world of system design.
A misconception I see regularly with new SAs is that they think their job is solely to come up with a way to solve a business problem. While that is fundamentally correct, there’s a whole lot more to it than that.
While a race car driver’s job is to drive a car around a track, their objective is to do it faster than everyone else. A solution architect’s job is to design systems, but their objective is to do it at the lowest possible cost.
Understand Python Lambdas in 3 Minutes
In Python, a lambda function is an anonymous function (a function without a name) that can take any number of arguments but only contains a single expression. As an example, here is a lambda that multiplies a number by three:
lambda x : x * 3
Today Big Tech Earnings In A Mere 700-Words
Today was yet another day of earnings from tech’s biggest names. To keep you up to speed without burying you in an endless crush of numbers we’ve pulled out the key data from each of the major reports.
In each, you will also find a link to their earnings reports. What does all of the data from the week’s earnings downloads mean for startups? We’ll have a full roundup on that front tomorrow morning, so stay tuned.
End Of Day Six
Whether you are looking to pick the right software service or you are wondering what exactly the day-to-day life of a data engineer is like.
I have the answers or at least the perspective.
Thanks so much for joining us for this week's newsletter.