Becoming A Better Data Engineer - Tips On Translating Business Requirements

One of the mistakes you’ll make as a data engineer or data scientist early in your career is not truly understanding the business requirements.

The business will come to you and ask for a real-time dashboard.

But they mean they want the data updated 3-4x a day, or maybe they only look at the report once a week; at that moment, the data should be as up-to-date as possible.

The business will ask for a machine learning (ML) model to help detect fraud.

But once you understand what the business generally knows is fraud, you'll realize you just need basic anomaly detection.

This isn't to say you don't ever need more ML models or never need to deploy a near-real-time system.

But it's good to figure out what the business really needs before building what they describe.

In this article, I wanted to discuss some tips that all engineers and technical employees can benefit from when it comes to gathering business requirements.

The Problem - The Business Doesn’t Always Know What They Are Asking

There is a classic sketch comedy bit that covers the gap between the tech domain and the business world. Of course, they simplified concepts down to “red-lines,” but it covers the problem well.

The issue isn’t that the business side isn’t intelligent; it’s that they don’t work in data (That’s well, the data teams job!). They generally know the terms and big picture but might not understand how all the pieces fit together on a nuanced level. Although I have met plenty of directors and CEOs who are very capable of going toe to toe in terms of discussing technology. But at the end of the day, they don’t have time to solve technical problems.

Add in all the management consulting companies telling them they have to jump on LLMs or Big Data because they need to sell services, and well…of course, they come to the data and tech teams asking what is being done to help them, the business, keep ahead of their competitors.

At the end of the day, that’s their goal. They don’t need to spend time learning the intricacies of how to design a data warehouse or develop an ML model, or even where it’d be best to fit said model. That’s the tech team's job.

But another goal of tech and data teams is to help guide the business in what is actually useful. I can’t tell you the number of conversations I have had where I had to convince clients that ML was overkill. Sure, machine learning can prove to be useful, but it can also be gold-plating. I get it, we all want to say we “added AI” into our products.

But not every problem requires it.

In the end, in order to understand and provide guidance for the business, we must first ask a lot of questions.

Ask Why, a Lot - The 5 Whys

One of the mistakes I made early on in my career was not asking why enough.

Perhaps it was the assumption that the people making requests knew precisely what they were asking for, or maybe I needed to just get started on work.

This can lead to a lot of incorrect assumptions.

In turn, that’ll lead to the wrong end state of your project.

Much of which can be mitigated by asking Why among other questions. In fact, many call this process, “The 5 Whys”. The number itself is less important, the goal is to really dig into the root cause of the ask.

You need to actually understand what the business is planning to do with the results of your project. If it’s a dashboard, you can ask:

• Who will be using this dashboard?

• How will it help them in their day to day?

• How often will they look at it?

• Is this dashboard or report for a specific meeting(often there is a monthly or weekly meeting when the numbers will be reviewed)

All of this can provide greater context on how you should actually develop your system.

Furthermore, you’ll be asked to add all sorts of features like real-time data, machine learning, and LLMs, all of which may or may not be necessary. However, you won’t know until you ask.

But don’t ask, “Do you need real-time data?”

The answer will be yes.

Asking broad questions that don’t make it easy for the end-user to actually answer or that don’t really clarify what an end-user is saying don’t help. They already told you they want real-time data. Now how can you get them to specify what they mean?

But those things need not be a 100-page requirements document.

In fact, this kind of leads into tip #2.

Understand The Business - Not Just The Technical Requirements

Don’t just ask technical questions because you want technical answers. That skips the “business” part of “business requirements”.

Ask questions that help you understand the business. You should focus on trying to understand what your stakeholders do, not just with the data or dashboards you’re putting together. But also what is there day to day like.

First, being empathetic builds trust.

Second, it can help you understand why the stakeholder or end-user is asking for whatever data or dashboard they have requested. Perhaps you can recommend a simpler solution.

Or, on the other hand, perhaps the solution might be far more complex but the initial ask might not cover the entire scope. That’s why it’s worth spending time asking your end-user for more than just a bunch of requirements.

All in all, words, conversations, and questions can only get you so far. The end-users need to start to see real “things,” whether that be dashboards, diagrams, or even a CLI that performs a basic task.