SeattleDataGuy’s Newsletter

Join My Technical Consultants Community

If you’re a data consultant or considering becoming one then you should join the Technical Freelancer Community! We have nearly 500 members!

You’ll find plenty of free resources you can access to expedite your journey as a technical consultant as well as be able to talk to other consultants about questions you may have!

Articles Worth Reading

There are 20,000 new articles posted on Medium daily and that’s just Medium! I have spent a lot of time sifting through some of these articles as well as TechCrunch and companies tech blog and wanted to share some of my favorites!

7 Mindsets That Are Slowing Down Your Career Growth

Irina Stanescu

often hears from her clients: “I do all the right things, but somehow I still am not getting put up for promotions. Why is that?”. After digging deeper, she can usually spot what’s getting in the way, and it’s often related to a mindset issue.

In today’s article, we’re breaking down 7 of the most common mental barriers and reframing them into empowering alternatives so you can take your career anywhere you want it to (whether it’s a promotion or other pursuits).

Vendor-Driven Design - The Role Vendors Play In Shaping Data Best Practices And Standards

Jul 16

Vendor-Driven Design - The Role Vendors Play In Shaping Data Best Practices And Standards

What if I told you that many of the concepts you believe are best practices and standards in the data and tech world are lies? Ok, lies is too strong of a word. But they are heavily influenced by vendors. Don't believe me? Think about T-SQL and PL/SQL. For those who are new to the data world, those were Microsoft's and Oracle's proprietary extensions of SQL.

End Of Day 136

Thanks for checking out our community. We put out 3-4 Newsletters a week discussing data, tech, and start-ups.

Vendor-Driven Design - The Role Vendors Play In Shaping Data Best Practices And Standards

Jul 16, 2024

What if I told you that many of the concepts you believe are best practices and standards in the data and tech world are lies?

Ok, lies is too strong of a word.

But they are heavily influenced by vendors.

Don't believe me?

Think about T-SQL and PL/SQL. For those who are new to the data world, those were Microsoft's and Oracle's proprietary extensions of SQL.

How about the Data Lakes and managed Hadoop providers.

How about the idea of NoSQL replacing traditional databases and MongoDB(amongst others)?

I am not saying these best practices, tools, and standards are bad per se. After all, every one of the above referenced examples has their place. They came about for a reason.

But I do think it's important to understand if you're coming into the industry, that many of these ideas being pushed benefit specific companies.

And to be clear, we are all likely curious about the newest hyped words. No one is safe from it. It’s part of who we are as technologists and engineers. We want to be on the cutting edge. We want to know what is going on.

It can also distract us from just delivering what the business needs.

So to all you new data engineers and analysts, always read and take in ideas with a grain of salt and ask yourself if using any best practice or design pattern fits your use case and why the best practice even exists.

With that let’s go over an example and discuss how you can approach the issue of vendor-driven design.

Digging Deeper Into A Recent Example

A recent example of a term being pitched as a future architecture is the “Composable Data Stack”. I have been seen a few posts about it(which you can see one of them below) and for all extents and purposes, the term can feel like it has come out of no where.

So where did it come from?

Likely it stems from the paper put out by Wes McKinney, Pedro Pedreira, Satya R Valluri, Mohamed Zait, Orri Erlin, Konstantinos Karanasos, Jacques Nadeau and Scott Schneider all of who worked at several different companies including Meta and Databricks to write the Composable Data Management System Manifesto.

There are dozens of terms like this that are floating around the data and tech space just like it. It’s one of the many ways we test out new ideas to see if they could replace a prior methodology or approach.

I believe that one of the reasons that some of these terms and marketing take off is because they capture some zeitgeist or answer a problem that people have.

Actually, I really like how

Richard Millington

brought this up in a video he recently put together for the TFA community where he referenced defining clear terms that encapsulate the solution you are selling.

It’s part of the process. Companies, consultants, and individuals likely see a trend in how companies are solving a problem and then try to capture it.

They give it a name, and either with enough of a marketing budget, good timing, or likely a combination of forces. Suddenly everyone is using the term.

That’s a win.

The struggle occurs as the term naturally gets used more and more and starts to a lose a little bit of its meaning(if not all of it). As soon as other marketers and sales team see it, they will all try to shoehorn their product into the space.

Whether it makes sense or not and then we see the cycle below where a term goes from having meaning to meaning almost anything.

All that being said, I do think that problems can arise when you allow vendors to drive your design.

The Problem

Let’s talk about some of the issues you’ll face when designing your data infrastructure the way the vendor says you should.

Actually, let’s focus on just one example. The issue you’ll often face when you follow vendor-driven design is the vendor may be focusing specifically on the benefits their platform provides while ignoring the many lessons and actual best practices we’ve picked up over the past few decades.

An example of this was how many tools in the modern data stack were all focused on singular tasks and didn’t really talk to each other. Suddenly, many data teams had to use a similar approach to the days of old when you had to use Cron or Windows scheduler to manage jobs. You’d have one task that took 20 minutes, give a little bit of buffer, then set the next task, etc.

I was going to use cron time, but it seemed messy.

Maybe if you were really ambitious, you’d set up some sort of lite dependency management system that tracked what jobs were completed and on what day, etc.

That’s why solutions like Airflow became popular (as well as every internally developed solution that was similar).

Yet we gave up those benefits for the trade-off of having solutions that were really good at one thing (like extract or transform) but didn’t really talk to each other. So we’d have to spend time to now integrate three solutions into Airflow itself or ignore the problem and just go back to having different steps run and hope one completed before the next one started.

This eventually led to the need for tools like 5x, Mage Mozart Data, and Orchestra that all wrap around these tools to orchestrate the workflows.

Now there is an argument to be made that this is a better end state. Perhaps you like it.

Even there, that problem was essentially created by the vendor marketing push for unbundled solutions and the solution is more vendors.

It kind feels like the TV/streaming unbundling where we suddenly realize we might be going back to a cable-like solution.

Overall, as long as you understand the situation and what you’re actually trying to build in terms of data infrastructure you’ll be able to discern what patterns will be best for your use cases.

What Should You Do

As I said earlier, none of this is per se bad. But it does mean, you, as an engineer, director, or technologist, need to dig deeper. You can’t just lazily read about best practices and assume it’s what you should do.

Instead, you should build a solid base of knowledge that includes many perspectives. Some of which you should disagree with but having a broader understanding of the data space helps provide you with the tools required to build better architecture.

Here are a few resources that I believe are worth digging into.

I hope that as you read or watch them, you realize that perhaps some of the problems you’re having today have existed for years and some of the solutions you believe are new…also have existed for decades.

Note: Also, if you believe you have some articles that should be added to this list, let me know!

Think Before You Design

It’s interesting when you ponder the fact that maybe a lot of what you think is good design, is just marketing well targeted at initially a specific problem you have, and eventually that solution is marketed as the only solution.

Overall, I think the important takeaway from this article should be to think for yourself.

Ask why you’re building what you’re building.

Not just in terms of “Are you building the right thing for the business” but “Are you picking and building the right data infrastructure for the problem?” Or are you letting someone else decide for you?

Honestly, the fun part, for me, is the design. So why rush through it!

Thanks for reading.

Video Of The Week: Don't Lead A Data Team Before Watching This - 5 Lessons You Need To Know As A Head of Data

Join My Data Engineering And Data Science Discord

If you’re looking to talk more about data engineering, data science, breaking into your first job, and finding other like minded data specialists. Then you should join the Seattle Data Guy discord!

We are now well over 7000 members!

Join My Technical Consultants Community

If you’re a data consultant or considering becoming one then you should join the Technical Freelancer Community! There are tons of free resources you can access to expedite your journey as a technical consultant.

Articles Worth Reading

Managing up: 3 things I wish I realized sooner

Jordan Cutler

Your relationship with your manager is the most important one to get right.

Having a solid relationship with them:

Opens you up to more opportunities to support your growth
Gives you a strong advocate when promotion time comes
Makes you feel psychological safety

But you’re rarely ever taught how to make that relationship happen. You have to figure it out yourself.

Organizational Anti-Patterns For Data Teams

Mar 30

Organizational Anti-Patterns For Data Teams

As companies continue to implement data and automated processes into their workflows and strategies, they unavoidably set up processes that, in the short run, help them develop faster but often have negative impacts in the long run. You could call these organizational anti-patterns or process anti-patterns. But honestly, that's me just looking for some form of a buzzword to fit my article around(although I did find the term "

End Of Day 135

Thanks for checking out our community. We put out 3-4 Newsletters a week discussing data, tech, and start-ups.

Don’t Lead a Data Team Before Reading This

Jul 6

Breaking Down Data Silos: Will Iceberg Finally Let Us Create A Single Source Of Truth?

Jun 30, 2024

With the recent acquisition of Tabular, it’s hard for me not to wonder...

Will large enterprises really create a single core data layer around Iceberg?

I have seen companies where their data is centralized into a single core data layer for analytics(which I am saying data layer as a general term to cover data warehouse, lake house, mesh, etc).

But in particular, that was at companies like Facebook which had a mature set of processes in place to ensure engineers utilized the same technology tool sets. This made it really easy to use data from other teams. Not to mention the fact that it was very clear which table was “THE” table, like the User table at Facebook that everyone knew and was literally referenced in the onboarding.

This isn’t always the case in many large enterprises. Many companies must deal with multiple data analytics platforms, one for marketing, one for the UK office, one for the recently acquired competitor, etc. To add to the problem, in some cases, there is a struggle to even have a single agreed-upon customer table.

This is caused by:

Office politics
Regulations
Shadow data teams
A single engineer who thinks that using <Insert Technology> will be optimal

Now perhaps I am thinking about this too rigidly. I am just curious to see if we could actually get to a point where we get to that mythical "EDW" we all read about but rarely have seen.

And perhaps it’s time to throw off some of the limitations of needing to only use one solution like BigQuery, Teradata or Snowflake to get there.

In this article I wanted to outline some of blockers that data teams and organizations will have to overcome as they attempt to create a single core data layer for analytics.

Of course, as someone commented, the European Sales Data would be in SAP

Is It Possible?

Is the final state possible?

Yes.

Let’s just get that out of the way. I referenced my example at Facebook and I have seen other companies do it. Many who are traditional large enterprises.

But it takes a considerable amount of effort, and even once completed, there are always little exclaves of data that live here and there.

Politics and Silos

One of the common issues many large enterprises run into is company politics. If you’re the VP or department head, you’ll likely want to spearhead the new and cool AI, Big Data LLM initiative, especially if you just got hired on.

That way you can say you drove some level of impact that perhaps doesn’t even make sense.

Or as brought up by Ravi, you want to be the one that successfully runs the migration to the new cool technology.

Each of these VPs and department heads create yet another data silo. All so they can have control. I was part of one of these projects early in my career. Did several data warehouses and a data lake already exist.

Yes.

Did someone on our leadership team want to get control of their own so they could pick what projects were done.

Yes.

Now suddenly you have a 5th data silo with a lot of repeat data.

Of course, not all data silos are created due to politics. Some are due to regulations, others due to shadow data teams, and still others just come from a lack of communication between different departments.

Shadow Data Teams

Now as referenced above, data silos have multiple sources. Even if you do get a company on board with building a single core data layer, some team, somewhere, will want a data set before it’s scheduled to be deployed.

In turn, they’ll buy a low code solution that keeps them from having to bubble up to management and use that to build their own data layer.

What’s great is that nowadays, things are in the cloud. Meaning you don’t have to ask IT to set up a server or manage anything. You can just spin up your own BigQuery or Snowflake instance and “Boom,” you have a “data warehouse.”

Now before you know it, you’ve got three customer tables each with different sources and different definitions of the term customer.

The Unspoken Problem of Data Silos

Here is the problem with data silos that goes beyond just the fact that data is partitioned by different tooling.

Even if you were to have federated queries, that can run across multiple data platforms. You run into the issue of every team creating their own versions of the same tables. A company might have five different customer tables, six sales tables, and three general ledger tables that all might be updated at different rates and might have different sources.

Some will include all data and some might just include data from specific entities; they all could be named in such a way that you can’t tell them apart.

This is the other side of attempting to create a single core layer of data (when I say that, I mean the base building blocks of data that represent your business, think customer, transaction, etc).

It’s not impossible but it won’t be fixed by technology alone, unless the technology makes it easy to fall in line and create very clear and concrete data models.

Security What I Am Really Scared About

Now you can manage security through various solutions such as Unity Catalog. But, you still have to actually do it. For example, below is a conversation coming from a post I made about having a single layer of data stored in Iceberg where one of the data analysts referenced not knowing how security would be implemented in a future project where they were doing just that.

My best guess is that the engineering team does know how security will be implemented. However, that is an assumption. If you plan to make a single core layer of data, it’s still important to maintain security and privacy. Not everyone should have access to everything. So part of your project should consider this as a crucial aspect.

What Will Likely Happen Next

I think my biggest point in this article is that companies won’t be able to switch to a single data layer overnight.

It’ll take a lot of communication, process and collaboration to get multiple department leaders on the same page.

I also believe that Iceberg has the potential to play a bigger role beyond just creating a single analytical layer. For example, Veronika posted their thoughts on how Iceberg might be used even for transactions or at least to make managing the process of getting data from transaction systems to analytical ones even easier.

But much of this is currently just theory(I assume some companies have something like this implemented somewhere).

Nevertheless, I am sure there are large consulting companies currently putting together presentations and white papers on how and why you should switch to a single Iceberg layer (some supporting Databricks and others Snowflake because, you know, those are the only two options). It might be the new meta that keeps consultants paid for the next 5 years.

So do watch out for new “best practices” and repackaging of old design practices. And make sure that whatever the future holds, you ask why you’re building what you’re building.

Technology Hasn’t Been the Only Issue

Rarely has technology alone been enough to change the way humans operate. I will add that if Iceberg or the future state of Tabular + Databricks makes it easy for engineers and analysts to do the right thing, then yeah, it’ll be more common to have a single core layer of data that represents a business.

Because then despite all of our politics, bad habits, and other internal company issues, the data layer might be able to push past it.

So is the ideal state possible? Yes. Will a lot of consulting companies make a lot of money from convincing companies that they should switch to this new future stay? Also yes.

With that, thanks for reading.

Complete Real-Time Stream Processing

The Power of Apache Flink in Minutes

Powered by Apache Flink, DeltaStream simplifies stream processing by eliminating setup complexities and the need for Flink expertise. Build materialized views and streaming applications with simple SQL and centralize all your data stores for a unified view and comprehensive insights. DeltaStream's flexible permissions model allows precise control over data access, enhancing team collaboration and security. With Role-Based Access Control, data governance is streamlined across the enterprise. Adapting to cloud-based or private SaaS deployment, DeltaStream's serverless structure eliminates cluster maintenance. Empower your team with real-time insights and innovative capabilities.

Interested? Check the link below for a free trial.

Get A Free Trial

Join My Data Engineering And Data Science Discord

If you’re looking to talk more about data engineering, data science, breaking into your first job, and finding other like minded data specialists. Then you should join the Seattle Data Guy discord!

We are close to passing 7000 members!

Join My Technical Consultants Community

If you’re a data consultant or considering becoming one then you should join the Technical Freelancer Community! I recently opened up a few sections to non-paying members so you can learn more about how to land clients, different types of projects you can run, and more!

Articles Worth Reading

Explaining Data Lakes, Data Lakehouses, Table Formats and Catalogs

In the past few years, open table formats have shaken up most of the data world. They have revolutionized how data lakes are managed which led to the Data Lakehouse. This article aims to explain key aspects of open table formats, their evolution, and their impact on data lake ecosystems.

We will take a look at leading formats like Apache Iceberg, Delta Lake, and Apache Hudi, providing insights into their technical methodologies and implementation strategies and we’ll also discuss how catalogs fit into the lakehouse architecture to demystify some of the confusion and complexity around Snowflakes and Databricks catalog services; Polaris and the open source Unity Catalog.

Data Modeling Where Theory Meets Reality

May 26

Data Modeling Where Theory Meets Reality

Data modeling varies at different companies. Even companies that all say they are using dimensional modeling or at least use terms like facts and dims might differ in the way they implement it. Over the past near decade, I have worked for and with different companies that have used various methods to capture this data. I wanted to review some of the techn…

End Of Day 133

Thanks for checking out our community. We put out 3-4 Newsletters a week discussing data, tech, and start-ups.

When and Why to Automate: A Data Engineer's Perspective

Jun 20, 2024

There has been a post going around LinkedIn that goes something like:

"Are you really a programmer if you haven't spent 40 hrs automating a 2 min task.”

I wanted to add my own spin on it so I added in:

Have you really worked in data if you haven't spent 40 hrs automating a 2 min task. On the flip side, many experienced data engineers and analysts will find a task taking 100s if not 1000s of hours a year and spend 10 hours automating it.

I had purely meant this as a point about growing as an engineer, meaning you learn how to find more valuable projects. But it brought in a lot of great perspectives, such as the one below.

Although hours saved are often a focal point of many automation projects, there are other reasons you might decide to automate a workflow.

So I wanted to discuss some of the reasons we automate tasks, as well as some of the reasons you shouldn’t.

The goal of this article is to:

Outline why we automate certain tasks
Call out some of the reasons not to automate a project
Have you, the reader pause and ask, should I really automate this?

With that let’s dive in to why you’d look to automate a task.

1. To Save Time

If you've been in tech for any amount of time, you’ve likely seen one of the many xkcd automation comics.

The one above is the one I think of most often when I consider the “Should I automate this?” question.

Maybe it’s a little “overly analytical”, but I do think if you only had one metric you could use, you’d likely want to pick this one.

Because one way or another, many automation projects come back to time saved.

Saving time in terms of:

Performing a task
Learning how to do a task(If you don’t know how to do a task, you can learn something else with your free time)
Fixing errors from a manual task
Having to replace someone who left because of doing a boring manual task

It really goes without saying. It also fits well into the mantra one of my managers told me, which was:

If you want to get promoted, don’t complete 1000 tasks, but instead, remove the need for 1000 tasks.

However, I did really like the annoyance metric brought up by Wouter(perhaps we should come up with a more technical term).

2. Morale Improvement(Getting Rid Of Annoying Tasks)

As pointed out by Wouter, some tasks are annoying. They bring down morale. People might even quit over it.

Now I’d argue that if you have an annoying task that only takes a minute but for some reason would take 100 hours to automate, it’d take a lot in terms of frequency to rationalize automating said process.

That aside, I have automated several processes that did fall into this category where the reason for automating was less about hours saved and more about trying to keep employee morale high.

For example one reason I have automated a process in the past was to ensure that people didn’t have to work weekends to get some numbers to executives. The reports themselves didn’t take long to run but on the flipside, the board reviewed the numbers every day, including weekends, which good for them; they were making plenty of money. They can work weekends if they want.

But it really wasn’t great for the IT team who only had one person who knew how to create the report.

Another similar process required an employee to wake up at 3 AM every morning to prepare a logistics schedule(this one also took several hours so it fits into the time category as well).

Now there are other tasks that are annoying that I am sure Wouter was referencing. I know plenty of people who take an hour to automate tasks that they just don’t want to do, and I get it!

But eventually, you’ll find that a task isn’t worth automating even if it’s a little annoying.

3. Out of Sight, Out of Mind

My Sprinkler Automated Control Panel From The 1980s

Automation is meant to help save time, yes, but also I like the idea of saving brain space. Both in terms of needing to think about a task as well as possibly ever needing to learn how to do said task to begin with.

For example, let’s talk about some fascinating technology. My sprinkler system.

It allows me to forget about watering my yard; it just magically happens every few days. This is more of the former of the two points referenced above as there isn’t too much to know about watering your yard(or so I thought until I recently asked someone to take care of some plants while I was gone on a trip).

Now refocusing on the business, there are tasks that if you could remove them, especially from a large set of businesses, you’d likely remove the need for humans to even know how to do said task. In other words, we can then spend our time learning new things and skills that could help a business grow in other ways.

In the end, saving brain space and processing.

4. Error Prone

One of the reasons many of my clients ask for a task to be automated is to reduce the error rate. Like anything, there is a spectrum of error rates and weight that could occur.

But I’ll break it down into two key reasons I have automated error prone workflows.

Small Repetitive Errors - Tasks done constantly throughout the day are likely very error-prone. For example, you’d think no one would make an error when typing in state abbreviations but they will.
Errors With High Costs - There are other tasks such as testing code, especially for a large company like Netflix or Facebook where a bad code launch could lead to millions if not billions of dollars in losses. Having an automated process ensuring that a task is done 99.99999% correctly vs. even a human’s 98% would be far more beneficial, even if the task itself only takes a few seconds.

In this way, reducing errors, whether frequent or infrequent, may be worth it. Also, if you think there are more reasons to automate due to error prone workflows, feel free to share them below!

Automation Gotchas

“The first rule of any technology used in a business is that automation applied to an efficient operation will magnify the efficiency. The second is that automation applied to an inefficient operation will magnify the inefficiency.” — Bill Gates

Now just because you can automate something doesn’t mean you should. Rube Goldberg’s drawings are an excellent example of what can happen when we try to automate/systemize everything without thinking about the actual reason for automating.

To add to the point, we just can’t automate everything as Leonardo puts it below.

In addition, as I discussed in my article about low-code and no-code, automating code writing, for example, could just lead to more bad code written faster. In the end, the goal is to simplify when possible and not to add unnecessary complexity in places where it wouldn’t actually benefit the end user.

Here are a few reasons why you might not want to automate a process.

Over-Automation: Automating tasks that are too complex or require human judgment can lead to inefficiencies and errors. Now likely some day in the future AI will be able to help discern some of these tasks, but I’d say we’re still figuring out where we can reliably implement said systems.
Lack of Flexibility: Automated systems can be rigid and may struggle to adapt to changes in the process or unexpected situations. If I had a nickel for every time someone told me an Excel or CSV would never change I wouldn’t need to work anymore.
High Initial Costs: The upfront investment in automation technology, software, and training can be significant. If you’re going to spend 1000 hours to save 10 hours a year, it’s likely a bad investment of time.
Maintenance and Upkeep: Automated systems require ongoing maintenance and updates. Not to mention if you build a heavily custom system that only one engineer knows how it operates, it’s not going to be good. That engineer will quit one day, the system will likely fail at some point and no one will know how to fix it. So maintenance and upkeep don’t just refer to when the current data team is maintaining it but also future engineers.

To Automate Or Not To Automate?

Overall, there are many reasons to automate a task, but not everything can or should be automated. I mean, we’ve all automated someones manual Excel workflow once and deeply regretted it.

Early on in most people’s careers, it feels like a great idea to automate everything! But I do think you should slowly find a methodology and approach for picking the right things to automate.

I will say if you just have some free time, sometimes automating a boring task when you have a few free hours is worth it. Even if it doesn’t perfectly fit some framework for “If you should automate or not”.

Thanks for reading.

Complete Real-Time Stream Processing

The Power of Apache Flink in Minutes

Interested? Check the link below for a free trial.

Get A Free Trial

Join My Data Engineering And Data Science Discord

If you’re looking to talk more about data engineering, data science, breaking into your first job, and finding other like minded data specialists. Then you should join the Seattle Data Guy discord!

We are close to passing 7000 members!

Join My Data Consultants Community

Articles Worth Reading

How Uber Accomplishes Job Counting At Scale

Uber operates on a massive scale, facilitating over 2.2 billion trips every quarter. Deriving even simple insights necessitates a scaled solution. In our case, we needed to count the number of jobs someone had participated in while on the Uber platform, for arbitrary time windows. This article focuses on the challenges faced and lessons learned as we integrated Apache Pinot™ into our solution.

How I Left Facebook to Forge My Path in Data Consulting

Feb 22

How I Left Facebook to Forge My Path in Data Consulting

Hi, fellow data practitioners, directors, scientists, and all-around amazing readers 👋 ! The Seattle Data Guy here! Today’s article will focus on talking about what it takes to start your own independent consulting company. If you are considering starting your own consulting company soon and you’d like even more guidance, then you can check out the