Beyond Big Tech: The Reality Of Data Engineering Outside Silicon Valley

I’ve had the privilege of working at a mix of companies—big tech, start-ups, enterprises, and nonprofits. Each has given me a unique perspective on how different environments approach data: the infrastructure they build to manage it and how they ultimately use it.

I’m writing this because, while tinkering on a small side project, I was hit with a vivid reminder of just how frustrating it can be to work with data from certain industries. In fact, when I made a humorous post about it, someone called out the poor data model in the raw file I was using.

But that’s the thing: it wasn’t my model—that’s just how the data came.

Companies with heavy digital footprints often create data that’s easier to model down the line, which is a big advantage.

And working in big tech? It offers developers more than just a fat paycheck: better tooling, well-integrated systems, and large support teams can save weeks of hassle when you’re trying to figure out which data system is the source of truth.

With that in mind, let’s dive into what data engineering looks like outside of Silicon Valley.

1) The Benefits I Had In Big Tech

I once worked with someone at Facebook who told me, “The amount of work we get done in a day at Facebook would take several weeks anywhere else.” And honestly, they weren’t wrong.

It wasn’t just about having better tools (though that helped). It was about the resources, the culture of execution and alignment, and the relentless drive to remove blockers. At Facebook, if you spotted a problem and came up with a solution that raised all boats, you’d be rewarded for it.

But the benefits went deeper than that. Here’s what really stood out:

• Optimized Infrastructure: Everything—from data pipelines to deployment processes—was built for scale and speed. Many companies I have worked for have manual deployment processes that are often cobbled together, and it was always clunky.

• Focused Teams: In many companies, you’re the data platform engineer, the data engineer, and the business analyst. In big tech, you get to focus on being a data engineer and mastering your craft.

• High Alignment: Teams shared a clear understanding of goals and priorities, reducing the need for rework or clarification.

• Investment in Data: Data wasn’t treated as a byproduct—it was a product in itself. That kind of investment created a level of rigor and innovation that’s hard to match.

This combination of infrastructure, focus, and talent created an environment where getting things done wasn’t just easier—it was inevitable.

2) The Reality of Companies Outside Big Tech

The reality of working in companies outside of big tech is that data is rarely the primary focus. No matter how much a company wants to prioritize it, if their bottom line doesn’t depend on data, justifying the spend can be tough—unless they have a strong data leader who can help leadership see the value.

This creates several challenges, including:

Cobbled-together data systems

In many companies, especially those shaped by mergers and acquisitions or internal politics (think VPs arguing over who gets to lead the AI project), data systems are often a patchwork of tools and platforms. This is true for both enterprises and smaller organizations, where it seems like they’ve thrown in just about every data tool imaginable.

It’s not uncommon to see companies using BigQuery, Snowflake, Databricks, and several self-hosted databases—all as part of their data warehouse solutions. Add in contracts with tools like Informatica, a self-hosted Airflow setup, and a custom-built data pipeline, and you’ve got a Frankenstein-like data stack.

To some degree, this is inevitable. Even when I worked at FB, a team member pushed to bring in Metabase—despite us already having two dashboarding solutions. Adding a third didn’t make much sense, but people like the tools they like, and they’ll fight to get them approved. Worse, if they don’t, a shadow data team will be created.