Kickstart Your Data Engineering Career: 9 Resources and Templates You Need

🎁 Plus an Additional 4 More For Future Data Consultants!

Back in 2019, I put together one of the first resources I’d share with the data engineering community, an interview guide for data engineers.

Since then, I have shared over a dozen guides, templates, and resources to help data leaders, engineers, new consultants, and people just getting started in the industry.

I wanted to compile those resources and templates all in one place. So that way, whether you’re just looking to break into data engineering or you just got hired, you’ve got a reliable list of resources you can turn to!

So, let’s dive in!

Just Getting Started

If you’re just getting started in your data career, or perhaps you’re about to go out and start interviewing again, then the next four templates and resources are for you.

1) 100 Day Data Engineering Crash Course

There isn’t any easy path to becoming a data engineer. I still always am learning new things. However, I did put together a 100-day crash course with free content that you can use to kick-start your journey. It covers everything from the basics like SQL and programming to topics like the Cloud and Docker.

So, if you’re looking to start your journey, you can check it out below!

100-Day Data Engineering Crash Course

If you go far enough back, you’ll find my original data engineering roadmap. Actually, someone just tweeted at me about it. And I agree, it was a little haptic! That’s why when I went back and did the 100-day crash course I put it in a google sheet so it’d be far easier to follow.

2) Data Engineering Side-Project Idea Template

There are a few videos and articles that cover data engineering projects.

For example you can look into Joseph Machado’s End-to-end data engineering project - batch edition or Darshil Parmar’s Uber Data Analytics | End-To-End Data Engineering Project.

But at a certain point, you need to come up with your own project. You don’t want to be the 500,000th person to build a project and put it on your resume. You want to come up with your own projects.

That’s why I put together a guide to help you plan it out.

Data Engineering Side Project Idea Template

Also, if you want a little more depth to this guide, then you can check out this article.

3) Data Engineer Interview Study Guide

So, it’s time to start interviewing. It can be hard to know what to study.

Here's a quick bit of advice before we get to the actual resource. Make sure you ask the recruiter what to expect, and don’t start studying after you get the interview. I have plenty of people reach out and tell me they have two weeks before an interview, and they aren’t sure what to study. Honestly, you’re cutting it too close to not know.

That’s one reason I put together the guide below. To help people prepare for various interviews, whether on Amazon or Facebook.

But again, always ask your recruiter before diving into this guide!

Data Engineering Interview Study Guide

4) Resume Template

Resumes don’t have to be put together by a graphic designer to get you hired, but you do want to have some formatting and organization. Here is the resume I used that got me interviews at Facebook, Amazon, and several other companies.

That being said, I would always recommend you find a friend to review your resume after you put write it out. Specifically, I would have a friend who has a good sense for resumes(I have found most people know someone like this). If you're like me, you sometimes forget to include some of our own accomplishments or make our bullets too generic, and having a second pair of eyes helps.

Resume Template

Onboarding

Now once you do get hired, you want to make sure to put your best foot forward. That’s why I put together guides to help you, a new hire, onboard fast whether you’re an IC or a data leader.

5) Onboarding Guide

In my experience most companies like asking questions like, what do you plan to get up to speed in the first 90-days. So why not put together your own plan and make sure you track what you got done in those 90-days, because you will forget!

30-60-90 Day Checklist

6) Onboarding Guide For Managers

Now let’s say you’re a new director a manager and you just got hired to lead a data analytics team, where should you start? I also created an onboarding guide for new leaders below.

30-60-90 Day Checklist For Data Leaders

Also, I have recently written several articles for data leaders that you should check out!

• Don’t Lead a Data Team Before Reading This

• Building Credibility As A Data Leader

• The Art of Execution: Making Things Happen as a Data Leader

Project Templates

Knowing a lot of data and technical skills is great. But if you can find projects that are worth doing and more importantly, get them done, those skills aren’t as useful. So here are a few templates to help you find projects as well as provide updates to your leadership so they feel in the loop.

7) Project Request Tracking Template

Earlier in this article, you saw the guide for side-projects, but now let’s talk about picking actual projects. Projects that can actually deliver value. Truthfully, this resource is leaking a future article. I am currently writing an article that will cover the topic of picking valuable projects. But here is a resource I’ll be putting with it.

Project Request Tracking Template

8) Project Kick Off Template

Once you have picked the project, a lot of data teams fail to actually let all interested parties, and perhaps even those who might not be in the know, well, know that the project has started. I recommend pushing off some sort of kick-off email, slack message, etc, in the correct channel or email group.

You’ve got to call your shots! You can use this template here.

Project Kick-Off Template

9) Project Update Template

Guess what? After you write the kick-off email or Slack message, you shouldn’t cease all communication. It’s far better to keep people in the loop even if the updates aren’t perfect. If you don’t, people will eventually start to ask, why aren’t we getting updates? Are things not going well?

So you might as well tell them what is going on, what has been completed, and if something is blocked, what you plan to do about it using the template below.

Project Update Template

Consulting Templates

Now for those that are interested in starting their own consulting company. I actually have several templates and guides. I won’t put a long explanation with each of these, but here are a few you can use if you decide to go down the route of consulting!

• 30 Day Checklist - Starting Your Independent Consulting Company

• Discovery Call Template

• Proposal Email Template

• New Client Onboarding Checklist

What Resources Are Missing?

I hope you’ll find this list of resources and templates helpful! But also, if you feel like there are more that should be put together, do feel free to let me know. I’d love to have this as my standing list of resources that I can send to anyone in the future who might need some help.

As always, thanks for reading!

---

Video Of The Week: Building Your Own Data Pipeline Tool From Scratch - Should You Do It?

---

Join My Data Engineering And Data Science Discord

If you’re looking to talk more about data engineering, data science, breaking into your first job, and finding other like minded data specialists. Then you should join the Seattle Data Guy discord!

We are now over 8000 members!

---

Join My Technical Consultants Community

If you’re a data consultant or considering becoming one then you should join the Technical Freelancer Community! We have over 700 members!

You’ll find plenty of free resources you can access to expedite your journey as a technical consultant as well as be able to talk to other consultants about questions you may have!

---

Articles Worth Reading

There are 20,000 new articles posted on Medium daily and that’s just Medium! I have spent a lot of time sifting through some of these articles as well as TechCrunch and companies tech blog and wanted to share some of my favorites!

---

How to Solve Data Engineering Problems

By

Daniel Beach

One thing I find myself doing these days (I am unsure how I feel about this), is teaching others to solve problems … Data Engineering problems to be specific. It’s not a hard stretch for most to imagine that what a person does at Senior+ software-type levels is just write good code all day.

I assure you, this is not the case typically.

I mean, if you are at the Senior+ level of anything it’s kinda expected that you will be able to write good code as needed. You will learn you don’t need to be the best, but you need to be above average to be effective. So, if it isn’t about good code all day … what it is all about?

Read More Here

Why Data Teams Need to Understand Metrics: A Look at Starbucks’ Comparable Store Sales

A key responsibility for any data team is to understand the core metrics driving their business. Starting from the top, these metrics often include figures like gross revenue and expenses. However, these high-level metrics can feel too far removed and abstracted from the actual business. 

Many companies, therefore, break down these top-line metrics into more specific, easy to understand(in terms of how they impact the business) ones that collectively build up to key business goals. These metric trees reveal deeper layers of context, connecting individual data points to overarching metrics. The further you dive, the more insight you gain into the underlying factors that shape each metric. 

In this article, we’ll explore a real-world example of one of a real company’s key metrics, and how data teams could approach better understanding how to be strategic. 

Read More Here

---

End Of Day 151

Thanks for checking out our community. We put out 3-4 Newsletters a week discussing data, tech, and start-ups.

Data Sharing in the Real World: Why SFTP Remains Essential for Companies

Data Engineering Basics

Every day, companies share data sets of users, patient claims, financial transactions, and more with each other.

Most people might assume this would be via API. However, companies have been sharing data for decades using CSVs, TSVs, positional files, and other formats you might not be familiar with. Not via API, but SFTP.

I know I wouldn’t have guessed that’s how companies send data back and forth when I was in school.

If you’ve been in the industry for a while, you’ve probably come across automated SFTP jobs that do just that. You’ve also likely had to encrypt or decrypt a CSV and had to interpret a schema file with parsing instructions that—somehow—are always a bit off the first time.

Now, sure, we all want to build real-time systems that use LLMs and other flashy new tools and solutions. Sometimes, that’s not what is called for.

After all, companies of all sizes—even tech giants like Facebook and Airbnb—still use SFTP to share critical information for analytical purposes(as well as operational). So, let’s dig into what SFTP is and how you will likely work with it.

What Is SFTP?

For anyone unfamiliar, here’s a quick overview: SFTP stands for Secure File Transfer Protocol, a method companies use to safely send and receive files over the internet. Think of it as a secure digital mail service, where data files—like documents or CSVs—are the "packages."

SFTP is commonly used to transfer data files such as CSVs, TSVs, and pipe-delimited files to a server. From there, a job triggered by a scheduler or specific event retrieves the file and loads it into an external or internal database.

Why Do Companies Use SFTP?

Extracting data from APIs isn’t always the best option. Setting up an API takes time and resources to configure and maintain, while a basic SFTP job is considerably more straightforward to get up and running. 

Here are just a few examples where I have seen SFTP jobs set up:

• Aggregating and analyzing data and creating reporting on data such as healthcare claims.

• Partnering with another organization to deliver a service to customers.

• Coordinating with a third party to send physical mail (more common than you might think).

• Sending payroll files from stores or franchises to a centralized payroll system.

In fact, I have now worked with several companies that use SFTP because they centralize similar data sets from dozens, if not hundreds, of companies to create reporting and analytics. Essentially, they act as a central hub, as pictured below.

From there, the various files from the companies would be processed into their similar entities. For example, insurance claims data is often broken down into patients, eligibility, claims and RX at the very least. This allows you to start to build data products that range from algorithms to dashboards.

One of the points that is implicit here is that unlike the company sending the data, the aggregator has the benefit of now having multiple companies data on the same entities. Meaning you can provide a very unique value add which is bench marking. Because you can now help your customers of your data product better understand where they stand compared to their competition(assuming they are open to their data being used that way of course).

All of which starts with getting the data via SFTP.

But what do you need to consider when setting up SFTP jobs?

Setting Up Your SFTP Job

Extracting data from files via SFTP isn’t terribly complex. In fact, some might consider it boring and not as exciting as building data systems that can manage exabytes of data. But there are plenty of companies that will need this work done, regardless of whether you like it or not.

Which is why I wanted to write about this topic.

So, to securely manage SFTP jobs, there are a few essential elements to keep in mind—starting with user authentication and encryption.

Authentication

Just like with APIs, user authentication is essential when working with SFTP. Here are two common methods (along with whitelisting):

• SSH Keys: SFTP typically relies on SSH keys for user authentication. Instead of using a password, you set up a public-private key pair: the server holds the public key, while the client keeps the private key securely. When you need to connect to the server you use the private key to authenticate.

• Password Authentication: This is exactly what you’d think. A username and a password.

File Encryption

Beyond authentication, many files transferred via SFTP are also encrypted using PGP (Pretty Good Privacy) Encryption. This is especially true when data has any form of PII, PHI or other forms of data that needs to remain secure.

There are a few ways a file might be encrypted, so it’s best to establish a standard approach with your external partners. Otherwise, you may find yourself juggling multiple encryption methods, which can be tedious when you write out your SFTP jobs (as I learned firsthand at Facebook). 

Some companies might use a key pair approach, others might use a password and still others might use a combination. Here are a few explanations of each.

PGP Key Pair Encryption

This is the most common PGP method I’ve encountered. I’ve used it when working with insurance claims data, HR data, telecommunications data, and more. Each of these had sensitive information that benefited from being encrypted on transfer.

PGP key pair encryption does this by using a public and private key for secure communication: the sender encrypts the file or message using the recipient's public key, and the recipient decrypts it with their private key. This also means you will have to learn how to create a public key pair. 

PGP Signatures (Signing Process)

PGP can also verify the authenticity and integrity of a file or message through digital signatures. This means the file will be checked to ensure you actually know who sent it. In this process, the sender "signs" a message or file using their private key, creating proof of origin. The recipient then uses the sender's public key to verify the signature, ensuring that the message or file hasn’t been altered and confirming the sender’s identity. I’ve used this for companies that were very concerned about possibly having a bad actor gain the ssh key or password and username information and send a file over that could contain malware or other malicious scripts or injections. Especially since the processes to extract the data are automated, most people likely don’t go back and check the hundreds of files they are receiving to see if they are malicious.

PGP Encryption + Signature (Combined Process)

Now, not finally in terms of methods that exist, but for the last example I’ll give, PGP can combine encryption and signing in a single process for added security. This means the message or file is both encrypted for confidentiality and signed for authenticity—a practice adopted by several teams I’ve worked with. 

Here’s how it works: first, the file or message is signed with the sender’s private key and then encrypted with the recipient’s public key. The recipient decrypts the message with their private key and verifies the signature using the sender’s public key.

Once you’ve got these two basics down, you can start considering the overall process and how you can ensure the data being processed is accurate.

Schema, Header, and Aggregate Files

When setting up an SFTP job, one of the first requirements is often a schema file. This file outlines the expected fields and their order, ensuring both sides have a common structure. Typically, companies prefer to define the schema to streamline integration on their end.

Especially if you are a data analytics company, the last thing you want to deal with is 100 different schemas that you now have to map. It doesn’t mean it doesn’t happen, of course. You should also set up a process where clear communication happens before said schema changes(cries in data engineering).

Once the schema file is agreed upon, there are a few additional files involved that might not have to do with the data, such as the header and aggregate files.

• Header: Sometimes, a header will exist in the data file, though this isn’t always the case. When present, the header allows the receiving party to check the validity of the schema. Because, guess what? On more than one occasion, another company changed the schema of the file they were sending me months after we had agreed to the schema.

• Aggregate File Checks: Less common but useful, an aggregate file is sometimes provided to double-check the integrity of the data in your main file. The file often includes an aggregate figure that describes the main file—like the total number of rows, total dollars spent, or unique users. This kind of acts like a Checksum, although it's usually a separate query that was run for the raw file.

Will We Ever Replace SFTP With Data Sharing

There are likely tens of thousands of SFTP jobs that are still running today. Who am I kidding, there are probably even more than that.

However, a concept like data sharing(which tools like Snowflake and Databricks allow for) could help nullify the need to create so many data copies—making both APIs and SFTP less necessary. 

With data sharing, companies could access shared datasets using just a few SQL commands, eliminating the heavy lifting involved in SFTP or API setups. That said, this shift is likely still 5-10 years away from becoming standard. 

How To Send Data Via SFTP

Until then, let’s look at what a basic script would look like to interact with an SFTP.

import os
import paramiko
import gnupg

# Setup for SFTP and PGP
SFTP_HOST = 'your_sftp_host'
SFTP_USERNAME = 'your_sftp_username'
SFTP_PASSWORD = 'your_sftp_password'  # Use None if using SSH key authentication
SFTP_PRIVATE_KEY = 'path/to/your/private/key'  # Set to None if using password
SFTP_REMOTE_PATH = '/path/to/your/encrypted/file.gpg'
LOCAL_FILE_PATH = 'downloaded_file.gpg'
DECRYPTED_FILE_PATH = 'decrypted_file.txt'

PGP_PASSPHRASE = 'your_private_key_passphrase'
PGP_PRIVATE_KEY = 'path/to/your/private/key'  # If not already imported into GPG

gpg = gnupg.GPG()

# If the private key is not imported, import it
with open(PGP_PRIVATE_KEY, 'r') as key_file:
    gpg.import_keys(key_file.read())

Connect to the SFTP server and download the file using paramiko
def download_file_from_sftp():
    transport = paramiko.Transport((SFTP_HOST, 22))

    # Authentication: Use either password or SSH private key
    if SFTP_PASSWORD:
        print("Authenticating with password...")
        transport.connect(username=SFTP_USERNAME, password=SFTP_PASSWORD)
    else:
        print("Authenticating with SSH private key...")
        private_key = paramiko.RSAKey.from_private_key_file(SFTP_PRIVATE_KEY)
        transport.connect(username=SFTP_USERNAME, pkey=private_key)

    # Start the SFTP session
    sftp = paramiko.SFTPClient.from_transport(transport)
    print(f"Connected to SFTP server: {SFTP_HOST}")

    # Download the remote file
    sftp.get(SFTP_REMOTE_PATH, LOCAL_FILE_PATH)
    print(f"Downloaded file: {SFTP_REMOTE_PATH} to {LOCAL_FILE_PATH}")

    # Close the SFTP session and connection
    sftp.close()
    transport.close()

#Decrypt the file using PGP key pair
def decrypt_pgp_file():
    with open(LOCAL_FILE_PATH, 'rb') as encrypted_file:
        decrypted_data = gpg.decrypt_file(encrypted_file, passphrase=PGP_PASSPHRASE)

    if decrypted_data.ok:
        with open(DECRYPTED_FILE_PATH, 'w') as decrypted_file:
            decrypted_file.write(str(decrypted_data))
        print(f"File successfully decrypted to {DECRYPTED_FILE_PATH}")
    else:
        print(f"Failed to decrypt file. Status: {decrypted_data.status}")

# Main flow
if __name__ == '__main__':
    # Download the encrypted file from SFTP
    download_file_from_sftp()

    # Decrypt the downloaded file
    decrypt_pgp_file()

SFTP Isn’t Going Anywhere

The first time I worked with SFTP wasn’t in school but at my second job. Before I knew it, I was managing similar workflows at Facebook and with 3-4 different clients.

While it may not be the most exciting work, chances are you’ll run into SFTP at some point, and you might even need to create a generic script that can handle the different authentication and decryption methods. Sure, there are some out-of-the-box options, but having a custom script gives you more control.

With that, thank you for reading!

---

Join My Data Engineering And Data Science Discord

If you’re looking to talk more about data engineering, data science, breaking into your first job, and finding other like minded data specialists. Then you should join the Seattle Data Guy discord!

We are now well over 7000 members!

---

Join My Technical Consultants Community

If you’re a data consultant or considering becoming one then you should join the Technical Freelancer Community! We have over 700 members!

You’ll find plenty of free resources you can access to expedite your journey as a technical consultant as well as be able to talk to other consultants about questions you may have!

---

Articles Worth Reading

There are 20,000 new articles posted on Medium daily and that’s just Medium! I have spent a lot of time sifting through some of these articles as well as TechCrunch and companies tech blog and wanted to share some of my favorites!

---

7 essential Mage features for data engineers in 2024

Data engineering continues to evolve causing developers to experience an array of old and new problems. Mage is a comprehensive data engineering platform that provides many solutions to help data engineers create pipelines rather than worry about managing infrastructure, handling complex integrations, and ensuring scalability. See the 7 essential features listed below and start developing pipelines.

Read More Here

---

End Of Day 149

Thanks for checking out our community. We put out 3-4 Newsletters a week discussing data, tech, and start-ups.

Should You Build a Custom Data Orchestration System? Here’s What to Consider

As you start building your first data pipelines, you’ll slowly realize you need to address a growing number of recurring issues. Maybe you implement a component or process that tracks what jobs are running, a scheduler, a set of generic scripts to run transforms and data ingestion, or even some form of UI.

Before you know it, you’ve pieced together something that looks like Airflow. Something that goes beyond just being a set of data pipelines but starts looking like an orchestrator.

Surprisingly (or maybe not), I’ve seen countless homegrown orchestration/data pipeline systems. Often, it feels like, given enough time, the team might build its own Airflow-esque solution.

In fact, I’ve come across plenty of posts where people ask if they should just build an orchestrator in-house.

That got me thinking: What does it really take to build a custom orchestrator and data workflow system? And should you even bother?

Here’s my take on what it involves, and a few examples from my past experiences. 

Examples of Data Workflow Systems I’ve Encountered

I’ve been fortunate to see a wide range of approaches to data pipelines and orchestration systems. Many teams don’t even have what you’d call a “workflow orchestrator.” Instead, it’s usually more of a script that calls a bunch of other scripts in a specific order.

So this would likely fit under the umbrella of a data pipeline or ETL/ELT system.

But let’s look at some real-world systems.

One common example of data pipelines is the standard script called by a scheduler. On Linux, it’s Cron; on Microsoft, it’s Task Scheduler. These systems often have a central script that drives the rest of the system. One particular instance I encountered relied on PowerShell instead of Python for the control system. But they all have similar components.

Here’s a breakdown of its key components:

• Transform Running Module

• Logging

• Data Quality Module

• Metadata Tracking

One example of a system I have seen is below.

This kind of setup is rigid, doesn’t necessarily track dependencies, and makes future changes less dynamic. Although it does allow you to at least just add new files to folders which is going in the right direction. Also, truth be told, I’ve seen even more rigid versions where every data connector is written in its own method and everything is very tightly written. It’s not exactly dynamic.

On the other end of the spectrum, companies like Airbnb, Facebook, and Spotify built their own orchestrators. Now there is a delineation between orchestrators and data pipeline systems. But functionally many companies will use these orchestrators to manage their data workflows.

If you haven’t seen Data Swarm, it’s very similar to Airflow; you can look at code examples below. 

Dataswarm code example from Mike Starr’s presentation. I will say this is rather old in terms of what dataswarm looks like. It’d be great to see if Dataswarm has changed since I last worked at FB.

These are built to be generalized so that any company can use them, covering key orchestrator facets like scheduling, dependency management, and UI. We’ll dive into these functionalities in more detail later.

Not to mention there is a broad set of alternatives to Airflow these days. They range from similar frameworks to low-code/no code options. The challenge comes when you realize that some of these alternatives are actually work flow orchestration solution and others are purely data pipeline solutions. This becomes apparent when you start to try to do work outside of the data warehouse/data lake.

For example you might want to push a file somewhere, manage external data connectors/transform solutions or perhaps refresh a data source for Tableau. Some of the tools on the spectrum will provide it, others won’t. Although, I am wondering if the below image to be more of a quadrant vs a spectrum.

After all, there are such a broad range of options. You might want to use a tool that is low-code/no-code or one that is just a Python library.

When Should You Build Your Own Orchestrator?

Let’s get to the part where I tell you if you should build your own orchestrator system. Obviously, there is a lot of nuance there. On the basic end, you might have a few Python scripts that handle extraction, transformation, and loading (ETL), all wrapped in a main script.

Since you’re only in need of a simple solution you might be able to get away with creating a basic data pipeline system. That isn’t an orchestrator but just a system that automates your basic data tasks.

On the flip side, you might be aiming to build a highly generic system that functions as the orchestrator itself. This is where you need to pause and ask yourself: Do you really want to reinvent Airflow? If you decide to pursue a generic solution, there are a few key considerations:

• Vision: If you plan to build a new orchestrator, I believe you need to pause and ask yourself a few questions. What will set your orchestrator apart from the many others out there? How will your solution be better? How are you reproaching the problem of managing automating tasks, specifically data?

• Design: You’ll face numerous design decisions from the outset. How will you handle dependencies? What’s your approach for loading new pipelines? If you start building your orchestrator without addressing these fundamental questions, you risk ending up with a solution that’s less effective than any of the existing data workflow orchestrators. Take the time to study past solutions—identify what worked, what didn’t, and why. Understand the core challenges of orchestration, then think critically about how you can design something better.

• Overhead: Building a successful orchestration solution is rarely a one-person job. While it might seem straightforward to replace all the functionality, even established orchestrators with large teams struggle to innovate and improve. Unless you’ve discovered a truly unique approach, it’s unlikely that a single engineer will be able to build a better solution than what’s already available.

• Adoption: Being a good engineer won’t be enough—you’ll also need to be a good marketer. Convincing engineers or other professionals to switch from familiar tools is like pulling teeth (just think of how entrenched Excel is). Without a solid go-to-market (GTM) plan, even the most well-designed generic orchestrator will struggle to gain traction and may go largely unnoticed. Even at your own company.

It’s important to note that I don’t mean to discourage you. However, if you plan on building your own orchestration layer, I hope you’ll take the points above into serious consideration.

Functionality Required For An Orchestrator

If you’re still hell-bent on building your own orchestrator and want it to serve as a general-purpose solution, here are some essential features to consider:

Backfill: The word “backfill” can send shivers down the spine of most data engineers. Much of this depends on how your orchestrator handles this task. You may need to re-run thousands of jobs, which can be time-consuming and error-prone. During my time at Facebook, I had to do this many times, and things didn’t always go smoothly. Streamlining this process is crucial for a great orchestrator.

Dependency Management: Even modern orchestrators, especially piecemeal solutions cobbled together from various tools, struggle with tracking which tasks have run and what should come next. Whether you’re dealing with real-time events or batch processes, your system needs to manage dependencies smoothly. Otherwise, you may end up with a tangled web of scripts that are difficult to maintain. 

Scheduling: In my three years at Facebook, daylight savings often cause scheduling headaches. Many jobs would end up running twice for the same hour, which in some cases might have caused issues. While our team took measures to avoid issues, other teams had to make adjustments to their pipelines. The reality is that something has to handle your task scheduling—will it be Cron or a more sophisticated solution? Whatever you choose, make sure it can handle these kinds of quirks without disrupting your workflows.

Metadata Management: There will be a lot of metadata that needs to be tracked—not just job dependencies, but also which jobs have been completed, their execution times, any errors they encounter, the types of errors, and job ownership. And that’s just the start. All this information needs to be organized and easily accessible.

Integration: There are plenty of great tools for handling transformations, extraction, data quality, and more. Your orchestrator needs to integrate well with these tools. Even Airflow has challenges when running dbt (and, of course, now dbt aims to be an orchestrator itself), so make integration a priority.

Alerting and Logging: Alerting systems often swing between being too sensitive or not sensitive enough—sometimes both at once. AI could potentially help address the problem of alert fatigue, where too many alerts cause important ones to be ignored. On the other hand, logging is generally more straightforward. As long as your system allows you to review past events and customize log messages, you should have what you need.

A Few Other Considerations

The above, in my world, would be considered the baseline. You should also consider how you’d handle the following:

• Data Governance

• Data Lineage

• Ease of managing different environments(Dev, Test, Production)

• Testing

• CI/CD

When you first start building a data workflow orchestration system, it might seem straightforward. However, as you add more functionality and scale, you’ll quickly realize that it’s much more complex than simply organizing a few scripts to run at specific times.

Note: Do let me know if you feel there are other must haves for a workflow orchestrator.

So Should You Build Your Own?

Whenever I see someone start building their own data pipeline system, I can’t help but feel a bit concerned. Don’t get me wrong, if you keep it simple, things will run smoothly for a while. But over time, as you build more and more pipelines, you’ll realize that a collection of scripts calling scripts quickly falls short. You’ll end up needing a generic logging system, a database connector, an API connector, and more. 

And don’t forget the inevitable schema changes and a whole host of other challenges that come with scaling.

I’m not here to completely discourage anyone from building their own solution—just make sure you’ve taken a hard look at all the available options and weighed the costs and benefits carefully.

I believe this comment below sums up my thoughts pretty well. If you do plan to go down this route. Don’t just think about the surface level issues that you’re facing with the current orchestrator. There is likely a reason why these solutions made certain design decisions.

So if you go a few steps deeper and real analyze the problem space, and see something that is worth reconsidering, and you want to commit the next decade building out your solution, then I’d say go for it!

With that, thanks for reading!

---

Join My Data Engineering And Data Science Discord

If you’re looking to talk more about data engineering, data science, breaking into your first job, and finding other like minded data specialists. Then you should join the Seattle Data Guy discord!

We are now well over 7000 members!

---

Join My Technical Consultants Community

If you’re a data consultant or considering becoming one then you should join the Technical Freelancer Community! We have over 700 members!

You’ll find plenty of free resources you can access to expedite your journey as a technical consultant as well as be able to talk to other consultants about questions you may have!

---

Articles Worth Reading

There are 20,000 new articles posted on Medium daily and that’s just Medium! I have spent a lot of time sifting through some of these articles as well as TechCrunch and companies tech blog and wanted to share some of my favorites!

---

Genie: Uber’s Gen AI On-Call Copilot

In today’s fast-paced tech environment(Tell me you used ChatGPT to help write this article without telling me you used ChatGPT to help write this article), maintaining robust on-call operations is crucial for ensuring seamless service functioning. Modern platform engineering teams face the challenge of efficiently managing on-call schedules, incident response, communication during critical moments, and strong customer support on Slack^® channels.

At Uber, different teams like the Michelangelo team have Slack support channels where their internal users can ask for help. People ask around 45,000 questions on these channels each month, as shown in Figure 1. High question volumes and long response wait times reduce productivity for users and on-call engineers.

Read More Here

---

End Of Day 148

Thanks for checking out our community. We put out 3-4 Newsletters a week discussing data, tech, and start-ups.