When and Why to Automate: A Data Engineer's Perspective

There has been a post going around LinkedIn that goes something like:

"Are you really a programmer if you haven't spent 40 hrs automating a 2 min task.”

I wanted to add my own spin on it so I added in:

Have you really worked in data if you haven't spent 40 hrs automating a 2 min task. On the flip side, many experienced data engineers and analysts will find a task taking 100s if not 1000s of hours a year and spend 10 hours automating it.

I had purely meant this as a point about growing as an engineer, meaning you learn how to find more valuable projects. But it brought in a lot of great perspectives, such as the one below.

Although hours saved are often a focal point of many automation projects, there are other reasons you might decide to automate a workflow.

So I wanted to discuss some of the reasons we automate tasks, as well as some of the reasons you shouldn’t.

The goal of this article is to:

• Outline why we automate certain tasks

• Call out some of the reasons not to automate a project

• Have you, the reader pause and ask, should I really automate this?

With that let’s dive in to why you’d look to automate a task.

1. To Save Time

If you've been in tech for any amount of time, you’ve likely seen one of the many xkcd automation comics.

Source

The one above is the one I think of most often when I consider the “Should I automate this?” question.

Maybe it’s a little “overly analytical”, but I do think if you only had one metric you could use, you’d likely want to pick this one.

Because one way or another, many automation projects come back to time saved.

Saving time in terms of:

1. Performing a task

2. Learning how to do a task(If you don’t know how to do a task, you can learn something else with your free time)

3. Fixing errors from a manual task

4. Having to replace someone who left because of doing a boring manual task

It really goes without saying. It also fits well into the mantra one of my managers told me, which was:

If you want to get promoted, don’t complete 1000 tasks, but instead, remove the need for 1000 tasks.

However, I did really like the annoyance metric brought up by Wouter(perhaps we should come up with a more technical term).

2. Morale Improvement(Getting Rid Of Annoying Tasks)

As pointed out by Wouter, some tasks are annoying. They bring down morale. People might even quit over it.

Now I’d argue that if you have an annoying task that only takes a minute but for some reason would take 100 hours to automate, it’d take a lot in terms of frequency to rationalize automating said process.

That aside, I have automated several processes that did fall into this category where the reason for automating was less about hours saved and more about trying to keep employee morale high.

For example one reason I have automated a process in the past was to ensure that people didn’t have to work weekends to get some numbers to executives. The reports themselves didn’t take long to run but on the flipside, the board reviewed the numbers every day, including weekends, which good for them; they were making plenty of money. They can work weekends if they want.

But it really wasn’t great for the IT team who only had one person who knew how to create the report.

Another similar process required an employee to wake up at 3 AM every morning to prepare a logistics schedule(this one also took several hours so it fits into the time category as well).

Now there are other tasks that are annoying that I am sure Wouter was referencing. I know plenty of people who take an hour to automate tasks that they just don’t want to do, and I get it!

But eventually, you’ll find that a task isn’t worth automating even if it’s a little annoying.

3. Out of Sight, Out of Mind

My Sprinkler Automated Control Panel From The 1980s

Automation is meant to help save time, yes, but also I like the idea of saving brain space. Both in terms of needing to think about a task as well as possibly ever needing to learn how to do said task to begin with.

For example, let’s talk about some fascinating technology. My sprinkler system.

It allows me to forget about watering my yard; it just magically happens every few days. This is more of the former of the two points referenced above as there isn’t too much to know about watering your yard(or so I thought until I recently asked someone to take care of some plants while I was gone on a trip).

Now refocusing on the business, there are tasks that if you could remove them, especially from a large set of businesses, you’d likely remove the need for humans to even know how to do said task. In other words, we can then spend our time learning new things and skills that could help a business grow in other ways.

In the end, saving brain space and processing.

4. Error Prone

One of the reasons many of my clients ask for a task to be automated is to reduce the error rate. Like anything, there is a spectrum of error rates and weight that could occur.

But I’ll break it down into two key reasons I have automated error prone workflows.

• Small Repetitive Errors - Tasks done constantly throughout the day are likely very error-prone. For example, you’d think no one would make an error when typing in state abbreviations but they will.

• Errors With High Costs - There are other tasks such as testing code, especially for a large company like Netflix or Facebook where a bad code launch could lead to millions if not billions of dollars in losses. Having an automated process ensuring that a task is done 99.99999% correctly vs. even a human’s 98% would be far more beneficial, even if the task itself only takes a few seconds.

In this way, reducing errors, whether frequent or infrequent, may be worth it. Also, if you think there are more reasons to automate due to error prone workflows, feel free to share them below!

Automation Gotchas

“The first rule of any technology used in a business is that automation applied to an efficient operation will magnify the efficiency. The second is that automation applied to an inefficient operation will magnify the inefficiency.” — Bill Gates

Now just because you can automate something doesn’t mean you should. Rube Goldberg’s drawings are an excellent example of what can happen when we try to automate/systemize everything without thinking about the actual reason for automating.

To add to the point, we just can’t automate everything as Leonardo puts it below.

In addition, as I discussed in my article about low-code and no-code, automating code writing, for example, could just lead to more bad code written faster. In the end, the goal is to simplify when possible and not to add unnecessary complexity in places where it wouldn’t actually benefit the end user.

Here are a few reasons why you might not want to automate a process.

• Over-Automation: Automating tasks that are too complex or require human judgment can lead to inefficiencies and errors. Now likely some day in the future AI will be able to help discern some of these tasks, but I’d say we’re still figuring out where we can reliably implement said systems.

• Lack of Flexibility: Automated systems can be rigid and may struggle to adapt to changes in the process or unexpected situations. If I had a nickel for every time someone told me an Excel or CSV would never change I wouldn’t need to work anymore.

• High Initial Costs: The upfront investment in automation technology, software, and training can be significant. If you’re going to spend 1000 hours to save 10 hours a year, it’s likely a bad investment of time.

• Maintenance and Upkeep: Automated systems require ongoing maintenance and updates. Not to mention if you build a heavily custom system that only one engineer knows how it operates, it’s not going to be good. That engineer will quit one day, the system will likely fail at some point and no one will know how to fix it. So maintenance and upkeep don’t just refer to when the current data team is maintaining it but also future engineers.

To Automate Or Not To Automate?

Overall, there are many reasons to automate a task, but not everything can or should be automated. I mean, we’ve all automated someones manual Excel workflow once and deeply regretted it.

Early on in most people’s careers, it feels like a great idea to automate everything! But I do think you should slowly find a methodology and approach for picking the right things to automate. 

I will say if you just have some free time, sometimes automating a boring task when you have a few free hours is worth it. Even if it doesn’t perfectly fit some framework for “If you should automate or not”.

Thanks for reading.

---

Complete Real-Time Stream Processing

The Power of Apache Flink in Minutes

Powered by Apache Flink, DeltaStream simplifies stream processing by eliminating setup complexities and the need for Flink expertise. Build materialized views and streaming applications with simple SQL and centralize all your data stores for a unified view and comprehensive insights. DeltaStream's flexible permissions model allows precise control over data access, enhancing team collaboration and security. With Role-Based Access Control, data governance is streamlined across the enterprise. Adapting to cloud-based or private SaaS deployment, DeltaStream's serverless structure eliminates cluster maintenance. Empower your team with real-time insights and innovative capabilities. 

Interested? Check the link below for a free trial.

---

Join My Data Engineering And Data Science Discord

If you’re looking to talk more about data engineering, data science, breaking into your first job, and finding other like minded data specialists. Then you should join the Seattle Data Guy discord!

We are close to passing 7000 members!

---

Join My Data Consultants Community

If you’re a data consultant or considering becoming one then you should join the Technical Freelancer Community! I recently opened up a few sections to non-paying members so you can learn more about how to land clients, different types of projects you can run, and more!

---

Articles Worth Reading

There are 20,000 new articles posted on Medium daily and that’s just Medium! I have spent a lot of time sifting through some of these articles as well as TechCrunch and companies tech blog and wanted to share some of my favorites!

---

How Uber Accomplishes Job Counting  At Scale

Uber operates on a massive scale, facilitating over 2.2 billion trips every quarter. Deriving even simple insights necessitates a scaled solution. In our case, we needed to count the number of jobs someone had participated in while on the Uber platform, for arbitrary time windows. This article focuses on the challenges faced and lessons learned as we integrated Apache Pinot™ into our solution.

Read More Here

---

End Of Day 132

Thanks for checking out our community. We put out 3-4 Newsletters a week discussing data, tech, and start-ups.