Becoming A Better Data Engineer - Tips On Translating Business Requirements

One of the mistakes you’ll make as a data engineer or data scientist early in your career is not truly understanding the business requirements.

The business will come to you and ask for a real-time dashboard.

But they mean they want the data updated 3-4x a day, or maybe they only look at the report once a week; at that moment, the data should be as up-to-date as possible.

The business will ask for a machine learning (ML) model to help detect fraud.

But once you understand what the business generally knows is fraud, you'll realize you just need basic anomaly detection.

This isn't to say you don't ever need more ML models or never need to deploy a near-real-time system.

But it's good to figure out what the business really needs before building what they describe.

In this article, I wanted to discuss some tips that all engineers and technical employees can benefit from when it comes to gathering business requirements.

The Problem - The Business Doesn’t Always Know What They Are Asking

There is a classic sketch comedy bit that covers the gap between the tech domain and the business world. Of course, they simplified concepts down to “red-lines,” but it covers the problem well.

The issue isn’t that the business side isn’t intelligent; it’s that they don’t work in data (That’s well, the data teams job!). They generally know the terms and big picture but might not understand how all the pieces fit together on a nuanced level. Although I have met plenty of directors and CEOs who are very capable of going toe to toe in terms of discussing technology. But at the end of the day, they don’t have time to solve technical problems.

Add in all the management consulting companies telling them they have to jump on LLMs or Big Data because they need to sell services, and well…of course, they come to the data and tech teams asking what is being done to help them, the business, keep ahead of their competitors.

At the end of the day, that’s their goal. They don’t need to spend time learning the intricacies of how to design a data warehouse or develop an ML model, or even where it’d be best to fit said model. That’s the tech team's job.

But another goal of tech and data teams is to help guide the business in what is actually useful. I can’t tell you the number of conversations I have had where I had to convince clients that ML was overkill. Sure, machine learning can prove to be useful, but it can also be gold-plating. I get it, we all want to say we “added AI” into our products.

But not every problem requires it.

In the end, in order to understand and provide guidance for the business, we must first ask a lot of questions.

Ask Why, a Lot (and Other Questions)

One of the mistakes I made early on in my career was not asking why.

Perhaps it was the assumption that the people making requests knew precisely what they were asking for, or maybe I needed to just get started on work.

This can lead to a lot of incorrect assumptions.

In turn, that’ll lead to the wrong end state of your project.

Much of which can be mitigated by asking Why among other questions.

You need to actually understand what the business is planning to do with the results of your project. If it’s a dashboard, you can ask:

• Who will be using this dashboard?

• How will it help them in their day to day?

• How often will they look at it?

• Is this dashboard or report for a specific meeting(often there is a monthly or weekly meeting when the numbers will be reviewed)

All of this can provide greater context on how you should actually develop your system.

Furthermore, you’ll be asked to add all sorts of features like real-time data, machine learning, and LLMs, all of which may or may not be necessary. However, you won’t know until you ask.

But don’t ask, “Do you need real-time data?”

The answer will be yes.

Asking broad questions that don’t make it easy for the end-user to actually answer or that don’t really clarify what an end-user is saying don’t help. They already told you they want real-time data. Now how can you get them to specify what they mean?

But those things need not be a 100-page requirements document.

In fact, this kind of leads into tip #2.

Understand The Business - Not Just The Technical Requirements

Don’t just ask technical questions because you want technical answers. That skips the “business” part of “business requirements”.

Ask questions that help you understand the business. You should focus on trying to understand what your stakeholders do, not just with the data or dashboards you’re putting together. But also what is there day to day like.

First, being empathetic builds trust.

Second, it can help you understand why the stakeholder or end-user is asking for whatever data or dashboard they have requested. Perhaps you can recommend a simpler solution.

Or, on the other hand, perhaps the solution might be far more complex but the initial ask might not cover the entire scope. That’s why it’s worth spending time asking your end-user for more than just a bunch of requirements.

All in all, words, conversations, and questions can only get you so far. The end-users need to start to see real “things,” whether that be dashboards, diagrams, or even a CLI that performs a basic task.

Visualize What You Can Mock Up and Iterate

Sometimes, the best way to get on the same page of what the business wants is to create visuals or make some form of tangible POC.

This in itself makes progress from words and abstract ideas to something that can be iterated on.

This doesn’t have to be a dashboard mock-up, although those can work well.

You can put together a Jupyter notebook and take your audience through a data story where they can gain greater clarity on what exactly they want to build in terms of features.

You can put together a diagram on Lucid Chart or Draw.io.

The point is these can act as talking pieces to help guide your business partners. For example, I recently worked with a client who wanted to build a data product.

In order to help guide the conversation on what we could put into a product, I analyzed the data in a Jupyter Notebook and then shared it with them. Now you could just shove a bunch of code in peoples’ faces or you could provide a well broken down analysis of what the data is saying. You can highlight were certain charts and parts of the analysis could be converted into either a dashboard or some other product and even reference the perceived value if you understand the business well enough.

That’s essentially what I did. In each section I highlighted how we could take their over-arching task into clear product modules and create real value.

Having a clear set of visuals makes complex ideas tangible,  in turn, turning the abstract into concrete final targets. This can also act as a final target your data team can shoot for.

Rather than constantly attempting to iterate off of written requirements, engineers and analysts can have a final state already pre-drawn out. 

From there, they can iterate closer and closer to the actual end goal.

Overall, gathering business requirements is not just about writing 100-page documents that, as Jeff Nemecek recently said on a live, no one will ever read.

It’s about understanding how your stakeholders operate and how your new feature, dashboard, model, etc, will help improve their lives.

Quick Pause: If your company is looking for advising on data strategy or data infrastructure, then feel free to reach out today for a free consultation(I promise I won’t tell you that you need an LLM)!

Understanding the Business  

There have been several articles I have shared in the past few months covering growing as an engineer.

• Getting Unstuck In Your Data Career

• How To Fast Track Your Career As An Engineer - by

  Ryan Peterman

• Becoming a Data Engineering Force Multiplier - By

  Daniel Beach

In each of these articles, there was some discussion of understanding the business. For anyone reading this that wants to grow in their career, there is some need to understand the business. Yes, technically, you must be very capable. 

You must understand the basics of whichever field you pick (data science, software engineering, data engineering, etc).

But you also need to take those technical skills and help the business understand what it's asking for. Sometimes, the business might want to invest in a $10 million project, and you may have a $1 million solution, but only if you understood why the business wanted to do the project in the first place. If you can bridge that gap between tech and business, there are many growth opportunities out there for you!

Thanks for reading.

---

Data Events

• Battle of The Ben(n)s And Data Stacks - With 5X - Virtual

• Sign Up For Data Engineering Things And SDG Data Conference - Late October - Virtual

---

Data Modeling With Joe Reis - Understanding What Data Modeling Is And Where It's Going

Articles Worth Reading

There are 20,000 new articles posted on Medium daily and that’s just Medium! I have spent a lot of time sifting through some of these articles as well as TechCrunch and companies tech blog and wanted to share some of my favorites!

---

12 Event-Driven Architecture Examples: Real-World Use Cases

Gone are the days when rigid, siloed application architectures couldn’t keep up with today's data-intensive business environments. Event-driven architecture (EDA) examples from leading companies show how industries are now adopting flexible, responsive systems to get real-time insights for their growth.

These architectures give a fresh event-driven approach so you can react dynamically to the ever-changing data flow environment. As the global microservices architecture market size is expected to reach $21.61 billion by 2030, companies are quickly catching on to the advantages of this approach to streamline their business processes.

If everyone else is benefiting from it, why should you miss out? What better way to understand their actual business impacts than exploring practical use cases across different sectors?

Read More Here

7 Data Engineering Projects To Put On Your Resume

Starting new data engineering projects can be challenging. Data engineers can get stuck on finding the right data for their data engineering project or picking the right tools.

And many of my Youtube followers agree as they confirmed in a recent poll that starting a new data engineering project was difficult.

Here were the key challenges they called out.

1. Identifying suitable data sets for the project.

2. Selecting the appropriate tools to employ.

3. Determining the subsequent steps once the data is acquired.

So to help inspire you I have collected several examples of data engineering projects that should help drive you forward.

Read More Here

---

End Of Day 93

Thanks for checking out our community. We put out 3-4 Newsletters a week discussing data, tech, and start-ups.