Becoming A Better Data Engineer - Tips On Translating Business Requirements

One of the mistakes you’ll make as a data engineer or data scientist early in your career is not truly understanding the business requirements.

The business will come to you and ask for a real-time dashboard.

But they mean they want the data updated 3-4x a day, or maybe they only look at the report once a week; at that moment, the data should be as up-to-date as possible.

The business will ask for a machine learning (ML) model to help detect fraud.

But once you understand what the business generally knows is fraud, you'll realize you just need basic anomaly detection.

This isn't to say you don't ever need more ML models or never need to deploy a near-real-time system.

But it's good to figure out what the business really needs before building what they describe.

In this article, I wanted to discuss some tips that all engineers and technical employees can benefit from when it comes to gathering business requirements.

The Problem - The Business Doesn’t Always Know What They Are Asking

There is a classic sketch comedy bit that covers the gap between the tech domain and the business world. Of course, they simplified concepts down to “red-lines,” but it covers the problem well.

The issue isn’t that the business side isn’t intelligent; it’s that they don’t work in data (That’s well, the data teams job!). They generally know the terms and big picture but might not understand how all the pieces fit together on a nuanced level. Although I have met plenty of directors and CEOs who are very capable of going toe to toe in terms of discussing technology. But at the end of the day, they don’t have time to solve technical problems.

Add in all the management consulting companies telling them they have to jump on LLMs or Big Data because they need to sell services, and well…of course, they come to the data and tech teams asking what is being done to help them, the business, keep ahead of their competitors.

At the end of the day, that’s their goal. They don’t need to spend time learning the intricacies of how to design a data warehouse or develop an ML model, or even where it’d be best to fit said model. That’s the tech team's job.

But another goal of tech and data teams is to help guide the business in what is actually useful. I can’t tell you the number of conversations I have had where I had to convince clients that ML was overkill. Sure, machine learning can prove to be useful, but it can also be gold-plating. I get it, we all want to say we “added AI” into our products.

But not every problem requires it.

In the end, in order to understand and provide guidance for the business, we must first ask a lot of questions.

Ask Why, a Lot - The 5 Whys