Why Your Data Infrastructure Migration Project Will Fail (And How to Succeed)
Hi, fellow future and current Data Leaders; Ben here 👋
Today, we are going to dive into migration projects and what can hold them back from succeeding.
But before we jump in, I wanted to share a bit about Estuary, a platform I've used to help make clients' data workflows easier and am an adviser for. Estuary helps teams easily move data in real-time or on a schedule, from databases and SaaS apps to data lakes and warehouses, empowering data leaders to focus on strategy and impact rather than getting bogged down by infrastructure challenges. If you want to simplify your data workflows, check them out today.
Now let’s jump into the article!
Intro
If you work in tech long enough, you’ll inevitably find yourself in the middle of a migration project—probably more often than you’d like.
“We should all use Teradata!” Migrate.
“Actually, Snowflake is the future!” Migrate.
“Wait, let’s go all-in on Iceberg and Databricks!” Migrate.
It’s a never-ending cycle. In fact, migration is such a core part of the industry that Databricks recently acquired BladeBridge just to help them make migrations easier.
The problem? Migrations often don’t go as planned. Some get delayed, others stall out halfway, and many become massive time and money sinks. There are countless ways for a migration to go sideways, but with the right approach, you can avoid the most common pitfalls.
Let’s break down the biggest reasons migration projects fail—and how you can keep yours on track.
Lack of a Plan
I’ve been involved in plenty of migrations—some smooth, some absolute nightmares. The difference? A solid plan.
You can’t just say, “Let’s migrate to Databricks!” or “Snowflake is the way to go!” and hope for the best (gotta be fair to both). Without a structured approach, things will get messy fast.
At a minimum, you need to:
List everything that needs to be migrated—tables, workflows and any other objects and artifacts.
Prioritize the order of migration—some dependencies need to move first.
Secure buy-in from beyond engineering—migrations impact more than just the data team.
You’d be surprised how many projects I’ve walked into where there was no clear plan. I’d ask around for a checklist of what needed to be migrated, and no one had one. Instead, people were trying to track everything in their heads—a guaranteed way to forget something critical.
So, start by listing out every object. Most databases let you access the information schema, so pull a list of all tables, stored procedures, tasks, and anything else relevant.
For dashboards, Excel reports, or other user-facing tools, you may need to manually gather details—unless you’re lucky enough to have a lineage tool. Truthfully, this is what made migration projects at Facebook much easier; I could easily list all the dependencies. Of course, coordinating across dozens of teams was still a challenge.
If you don’t have an automated way to track dependencies, get ahead of the problem. Ask every team that relies on your data warehouse to submit a list of what they need migrated—set a deadline, and hold them to it.
A little upfront planning can save you months of headaches.
Lack of Ownership
Some projects seem to start with a big announcement—“We need to migrate to XYZ platform”—but then… nothing. No clear owner, no defined process, just chaos.
Regardless of the project’s size or how many teams are involved, someone needs to own the migration. Without a clear leader, things fall apart.
I’ve been part of both heavily centralized and federated migrations(completely decentralized migrations tend to fall apart), and the right approach tends to depend on factors like company size, workload, and who originally built the workflows and tables.
In a federated approach, the data team (likely) leads the migration, setting standards and providing tools while other teams move their own assets.
In a centralized approach, one dedicated team handles the entire migration, reducing fragmentation but increasing workload.
I have also rarely seen a purely decentralized approach lead to success as it often leads to a few key mistakes:
No central team overseeing the overall migration.
No requirement for teams to create a list and track what needs to be migrated
No clear definition of “done done.”
No prioritization, so migration takes a back seat to daily work.
And that’s how a 6-month migration turns into a 2-year headache. Thus, having a federated approach works better for larger migrations.
Migrations are hard. They take time. And they can’t succeed without someone driving them forward.
Lack of Testing
Testing isn’t optional—it’s essential. It doesn’t matter if you’re moving from Python 2 to Python 3, Oracle to SQL Server, BigQuery to Snowflake(or the other way around), or whatever the new tool is that people want to move to—you need to verify that your data is actually the same post-migration.
I learned this when migrating from Oracle to SQL Server. We had a query that looked identical in both systems. Yet, when we ran it, we very quickly realized that the output was different. We had assumed null values would be sorted at the bottom, but SQL Server placed them at the top.
Same SQL statement. Different results.
This happens all the time. Teams assume that because they’ve copied the data, it must be a match—but different databases, ELT tools, and storage formats can introduce subtle changes. Like in the picture above where someone might have accidentally changed the data type and in turn altered the data.
And no, testing doesn't mean running a select * from query and manually reviewing each line.
Instead, use automated validation tools or write a simple script to compare tables across systems. If you don’t verify your data, you’ll be in for a nasty surprise when users start noticing discrepancies.
It can be tempting to think, “We’re just moving tables, the data will be fine.” But trust me—it’s worth double-checking.
Lack of Buy-in
A migration can be technically perfect and still fail—if no one actually wants to use the new system.
When it comes to migrations, you’re managing two key workflows:
Project Management
Ignore the second one, and you’ll find yourself in a bitter 6-12-month slog where teams resist the change, tensions rise, and people are thrown under the bus if things go wrong.
Migrations aren’t flashy. They don’t come with the hype of AI or the excitement of a new product launch. They’re tedious, often frustrating, and—if you don’t get buy-in from end-users and the decentralized teams who need to do the work—can lead to a never-ending cycle of complaints:
“Well, the old tool was better because it did…”
“This new system is just making my job harder.”
“I’ve had to get used to queries taking longer but it’s ok.”
Passive-aggressive comments and complaints can kill momentum fast. And, a great way to have a migration fail is to technically succeed but have no one buy-in.
To avoid this, involve end-users early—before the decision is set in stone. You don’t need universal agreement (because good luck getting even three teams to agree on anything), but you do need key stakeholders to feel like they’ve had a say. The earlier you bring them in, the less pushback you’ll face later.
Final Thoughts
Migration projects are unavoidable. Maybe your company needs new capabilities, or maybe a VP is convinced that a new tool will magically solve all your company's problems. Either way, a successful migration takes more than just an announcement.
Without a solid plan, clear ownership, thorough testing, and real buy-in, a migration can quickly spiral into frustration, delays, and—worst of all—an unfinished project that drags on for years.
The key? Put in the work upfront. Track progress, hold teams accountable, and don’t assume everything will just work itself out. A little planning now can save a lot of pain later.
Also, if you’re looking for a more in depth article on running a migration you can check out this article here:
Realities of Being A Data Engineer - Migrations
In 1981 Indiana Jones gave us the classic scene where Indy is trying to switch out an idol for a rock and when he fails he starts getting chased by a boulder.
Thanks for reading!
Video Of The Week - Build A Data Stack That Lasts - How To Ensure Your Data Infrastructure Is Maintainable
Join My Data Engineering And Data Science Discord
If you’re looking to talk more about data engineering, data science, breaking into your first job, and finding other like minded data specialists. Then you should join the Seattle Data Guy discord!
We are now over 8000 members!
Join My Technical Consultants Community
If you’re a data consultant or considering becoming one then you should join the Technical Freelancer Community! We have over 1500 members!
You’ll find plenty of free resources you can access to expedite your journey as a technical consultant as well as be able to talk to other consultants about questions you may have!
Articles Worth Reading
There are thousands of new articles posted daily all over the web! I have spent a lot of time sifting through some of these articles as well as TechCrunch and companies tech blog and wanted to share some of my favorites!
8 Guaranteed Ways to Annoy Your Senior Engineer
By
After 15 years of working hands-on as an engineer, I’ve worked with all kinds of people from various functions. Most interactions were great—people were smart and a pleasure to work with. But sometimes, they’d do things that would consistently grind my gears.
A while ago, I decided to share what annoys me on LinkedIn, and unsurprisingly, most engineers seem to be triggered by similar things.
This guide captures the most common ways folks unintentionally (or otherwise) drive senior (and above) engineers up the wall. Use it wisely!
If you’re an engineer, I’d love to hear which resonated most with you and which I missed. Also, maaaybe consider sharing them with the people who annoy you—maybe they’ll stop doing it, so all engineers can live happily ever after.
Data Sharing in the Real World: Why SFTP Remains Essential for Companies
Every day, companies share data sets of users, patient claims, financial transactions, and more with each other.
End Of Day 163
Thanks for checking out our community. We put out 3-4 Newsletters a week discussing data, tech, and start-ups.
That’s very true for data migration projects, another big factor is not buying in business users that lead to adaption failures to new platforms.
"If you don’t have an automated way to track dependencies, get ahead of the problem. Ask every team that relies on your data warehouse to submit a list of what they need migrated—set a deadline, and hold them to it."
Alternatively, look carefully at whether a metadata data catalog will address a good portion of the dependency analysis that you need, one-demand and in realtime. Depending on the technology and the catalog vendor, often a catalog alleviates a good amount of the documentation burden on dependency analysis and understanding end-to-end.