From Day One to 100: The Seattle Data Guy Journey in One Special Issue
This is my 100th newsletter(at least since I started adding the “day” notation at the bottom of each newsletter).
Since starting, the newsletter has grown to nearly 60,000 subscribers and has had millions of views!
Honestly, that’s a lot more than I ever expected. Sure when I wrote on Medium I had similar numbers. But having 60,000 people signed up to get this newsletter directly sent to them is another thing altogether.
This newsletter has opened so many doors in terms of meeting a lot of great people, whether it be creators, founders, software engineers, data engineers, directors and so many others!
It’s also been a great way to think through and crystalize much of what I have learned in the last near decade of working in the data industry. Whether it be data modeling, career growth or how to manage teams.
Most importantly, it’s been a way to share not only the lessons I have learned but also the lessons from others as well!
So in this newsletter I wanted to reflect back on these articles, some of their themes and make sure if you’ve happened to miss one of these articles in the past, that you can read it now!.
Data Engineering Career
Whether you’re just trying to break into data or you’re looking to grow in your career, I have written several articles(as well as had other authors write) on the subject of career. There is so much more to growing in your career than just keeping up with technology. In fact, most of the articles below cover the soft skills that you will need to grow.
Becoming a Data Engineering Force Multiplier by - As the landscape becomes increasingly data-driven, fueled by the rise of ML and AI, the importance of data engineering as a function continues to climb steadily year after year.
🔒 Becoming A Better Data Engineer - Tips On Translating Business Requirements - One of the mistakes you’ll make as a data engineer or data scientist early in your career is not truly understanding the business requirements.
How To Fast Track Your Career As An Engineer by - If you’ve worked for a FAANG or large organization, you have likely heard the term “impact” thrown around like the mythical holy grail. If you’re doing work that doesn’t drive impact, then you’re not getting promoted (or at least that’s what it feels like).
🔒 Getting Unstuck In Your Data Career - Whether you’re a data engineer, data scientist, or software engineer, there are situations you can get yourself in that hold you back. You just get stuck, and are unsure how to grow to the next stage of your career.
Breaking Out Of Tutorial Hell - If you work in tech, then you know the curse of tutorial hell. You either hear about a new technology or finally convince yourself you’re going to learn a new skill like SQL, Python or A/B testing. So you start up a Udemy course and buy a book or join a boot camp with every intention of finishing it.
🔒 What Does It Take To Get Promoted As An Engineer? - 60% of the respondents to my 2022 State Of Data survey aren’t confident they know what is required to get a promotion. This made me re-examine what I know about leveling and skill expectations.
I do want to give a special thanks to everyone that has helped support this newsletter. Whether it be by views, clicks, likes, shares, etc! So here is 50% off a year!
Technology, Tools, Best Practices And Definitions
There are far too many terms, technologies and best practices to keep up with. So I love putting out articles that can help break down, and simplify these concepts. Some of these pieces are deeper dives, but others will be higher-level so you can see where different pieces fit.
Behind the Rust Hype: What Every Data Engineer Needs to Know by
- Daniel in particular enjoys writing about the nitty gritty details of different technologies. In this article he goes over Rust, where he sees it playing a role in data engineering and some basics in terms of code snippets.🔒 Introduction to Presto/Trino for Data Engineers - Have you ever been confused about what the story is with PrestoSQL, PrestoDB, and Trino? Well let’s dig into the many faces of the technology and what’s going on.
What Is Change Data Capture - Companies continue to look for methods to gain near-real-time access to their data for analytics. Honestly, this has been ongoing since I got into the data industry a decade ago…and well before that.
Why is Polars All The Rage? by
- There has been a lot of discussion and interest in the data engineering world around several specific libraries and languages. One of these libraries would have to be Polars. Does it live up to the hype?🔒 Mistakes I Have Seen When Data Teams Deploy Airflow - Airflow remains a popular choice when it comes to open-source orchestration tools. But it’s difficult to set-up well, especially when you need it to scale, read about a few problems I have seen in this article.
Data Engineering Vs Machine Learning Pipelines by
- Data engineering and machine learning pipelines are both very different but oddly can feel very similar. Many ML engineers I have talked to in the past rely on tools like Airflow to deploy their batch models.🔒 Operational Data Stores Vs Data Lakehouses And All The Other Data Management Methods - There are so many other methods technical teams use to manage their data, especially as AI and machine learning are having another resurgence. I wanted to go over the various ways companies are managing their data.
🔒 OLTP vs OLAP - Transactions Vs Analytics - OLTP vs OLAP is a simple concept to explain but it’s importance doesn’t generally become clear until you start trying to analyze data at scale and becomes more complex. There are plenty of solutions that have trade-offs, some that are in between and its important to understand why you might pick one data storage system vs another.
Managing Data Teams - (Videos Included)
Technology is only one pillar of having a successful data team. You also need solid processes and people to ensure that you can use your data successfully. So I put together several articles and interviewed several directors and managers in the data world to understand how to better manage data teams.
🎥 The Product Development Life-Cycle For Dashboards - With Abby Kai Liu - This interview with Abby provided a lot of great tactical steps your team can implement today to improve the way you treat dashboards, personally, a must-watch!
🔒 Centralized vs Decentralized vs Federated Data Teams - An important factor in succeeding when working in the data field is how you set up your team. There are so many combinations of centralized, decentralized, hybrid, etc that I have seen that I wanted to discuss these various methods, their pros and cons, and why you might select one vs. another.
🔒 Onboarding For Data Teams - One of the major success factors I find for data teams is their onboarding process. An organized and concise onboarding process helps set new hires in the right direction from day one vs. having them wonder what they are supposed to do or where data lives.
🎥 How Data Analytics Teams Can Deliver What The Business Needs - In the data world figuring out what your company and stakeholders need is sometimes one of the largest challenges we face. Yes, there are plenty of cool new technologies and methods that the business wants to implement. But it's our job to cut through the fluff and deliver what they need.
Different Types Of "Data Engineering" Teams - We’ve talked about how to organize data teams, but what about discussing different types of “data engineering teams”? Should your data infrastructure be managed by your data engineers or do you need a data platform team?
🎥 How To Manage Data Teams Successfully - Asking A Director Of Data Architecture And Governance - This was an amazing conversation I had with Jeff Nemecek, Director of Architecture at The Walt Disney Company. He dropped so many nuggets of wisdom in this talk!
Tips For Hiring Junior Data Engineers - Hiring junior data engineers is tricky. Many data engineering topics aren’t taught in school. So if you ask a newly minted computer science or information systems major to design a data model, they’d likely not know what you were asking for.
🔒 Elevating Your Data Team: From Service Providers to Strategic Partners - How do you create a data strategy roadmap that focuses on what your team should be doing? Especially if you’re data team has found itself stuck in the data service trap. Simply providing data and dashboards when asked and never really becoming a strategic partner.
Articles You Didn’t Realize I Wrote
Not everyone may know this, but I started writing another newsletter and created the Technical Freelancer Academy. Both which focus on consulting. Although I take it from the angle of data analytics consulting. Really, all the articles below can be used for almost any skill set that demands consulting.
How To Get Clients As A Consultant - Marketing Tips - There are many different ways to get clients. You can be great a selling, networking, partnering with vendors and my personal favorite is market. Marketing in particular can help build a sustainable system that can take a longer time to get started but once started, can help keep your prospect pipeline full.
My First 6-Figure Consulting Project - After signing my first 6-figure consulting contract, you’d think I’d be excited. I was. But honestly, the most palatable feeling was not excitement…it was an impending dread.
How I Landed More Than 1 Million Dollars In Revenue For Seattle Data Guy - Let me save you a click, writing content helped me attract clients.
Levels You'll Experience On Your Data Analytics Consulting Journey - If you’re looking to start consulting, you don’t have to quit your job tomorrow. You don’t need to risk your financial security before understanding if you even like the work. Instead, you can slowly build up your skills and prospect pipelines so that you have a steadier business when you are ready to quit.
How To Kill Your Profits As A Data Analytics Consultant - There are a lot of ways to make money consulting and there are a lot of ways to lose it. Let’s go over a few that I have done on the lose it side of that line.
Collaborations
I have done a lot of collaborative articles. Many that aren’t underneath this section. The first major collaboration was with
, and even though that was last year included it below. But I have also had the chance to talk to a lot of data leaders and specialists and I wanted to share those articles below.What Is Data Engineering - In short, data engineers play an important role in creating core data infrastructure that allows for analysts and end-users to interact with data which is often locked up in operations systems.
🔒 How Does League Of Legends Deploy Machine Learning Models Into The Game? by Ian Schweer - In this post, I want to share with you some challenges in telemetry, player behavior, and shipping data in game.
🔒 How Do These 5 Data Leaders Drive Value With Data - In this post, I want to share with you some challenges in telemetry, player behavior, and shipping data in game.
Looking Past Data Infrastructure - How To Deliver Value With Data with Gordon Wong - I wanted to take a deep dive into people who have led and managed data teams, whether it be BI or data engineering. Some data teams are likely starting to feel the pressure to deliver actual ROI.
The Data Industry Past Present and Future
Technology is fascinating, and the ways technology is adopted, marketed, adopted, and used are also fascinating. The articles below are all focused on that subject. This includes the State Of Data Engineering articles where I reviewed over 400 teams' data infrastructure stacks.
🔒 A Decade In Data Engineering - What Has Changed? - The concepts and skills data engineers need to know have been around for decades. However, the role itself has really only been around for a little over 10 years, with companies like Facebook, Netflix and Google leading the charge.
The State Of Data Engineering - Part 1 - About a year ago I started a survey to analyze what companies data infrastructure looked liked. We had over 400 people respond, and here was the first part of the results.
🔒 The Challenges You Will Face When Data Modeling - Data modeling in real life requires you to fully understand the data sources and your business use cases, which can be difficult to replicate as each business might have its data sources set up differently.
What Is Query Driven Data Modeling - According to
, many companies no longer perform the first two and often just go straight to developing queries to support a one-off request. Thus we are likely not even using any form of star schema or even asking ourselves whether we are building a data mart or data warehouse.
Plans For 2024
In 2024 I want to continue to provide content you want to read! Whether that be more collaborations with authors or even more in-depth looks at specific tools. I am already starting to reach out to people who I’d like to work with and writing out topics I believe need to be covered.
But I want to hear from you! I’d love to get your feedback. If you have topics, collaborations, or other thoughts on what you’d like covered in 2024, please feel free to share them.
Thanks Again!
Again, thank you to everyone who has helped grow this community! I appreciate all your support and I look forward to growing this community even more in 2024!
Data Engineering And Machine Learning Summit 2023!
The Seattle Data Guy and Data Engineering Things are coming together to host the first Data Engineering And Machine Learning Summit on October 25th and 26th.
The purpose of this summit is to focus on the data practitioners who are actually doing the work at companies. Solving real problems with real solutions.
For example, here are just a few of the talks we have planned!
Sudhir Mallem - Centralized ETL framework for Decentralized teams at Uber
Jessica Iriarte - Optimizing oilfield operations using time-series sensor data
Kasia Rachuta - Overcoming challenges and pitfalls of A/B testing
Siegfried Eckstedt - Test-Driven Development for Your Data, Models and Pipelines
👩🏻💻 Mikiko Bazeley 👘 Bazeley - MLOps Beyond LLMs
Joe Reis 🤓 - WTF Is Data Modeling
You’re definitely not going to want to miss this event!
Also I want to give a very special thanks to our sponsors Decube, Segment, and Onehouse! Their support is ensuring we can make this conference even bigger and better.
End Of Day 100
Thanks for checking out our community. We put out 3-4 Newsletters a week discussing data, tech, and start-ups.