7 Lessons I Learned the Hard Way From 9+ Years as a Data Engineer
Reflection is an essential part of the learning process.
Learning, whether as a data engineer, analyst, or really any position, is not just about cramming as much knowledge into your head as possible; it’s about pausing to consider why certain knowledge matters and how it can be applied. Without that step, knowledge becomes a tool in your workshop that never gets used.
Sure, you can show off that cool tool or share quirky, one-off facts. It might seem impressive, but it doesn’t carry much weight. Taking time to reflect helps you understand where technical knowledge fits and what lessons you’ve genuinely absorbed over the past year, or even the last decade. Where have you made mistakes in picking the wrong data modeling processes or running the wrong project altogether? Many of these lessons have to be learned through time and making mistakes.
Here are some of the most important lessons I’ve learned about working with data and building effective systems.
As a quick side note, the idea for this article came from
similar articles for software engineers.1) Avoid Infrastructure for Infrastructure’s Sake
I first heard the phrase “infrastructure for infrastructure’s sake” from Ethan Aaron, and it really struck a chord.
Don’t get me wrong, I get it.
It's easy to get caught up in the allure of complex diagrams showcasing 12 different tools and services, wanting to replicate that for your data team.
I’ve encountered many projects where it was clear an engineer had built an overly elaborate system just for the sake of it, not to serve the business but to pad their resume. And to be clear, I have tried to do this as well. I was just lucky enough to work with more experienced data engineers who could help me understand why making certain design decisions weren’t going to be sustainable.
There is a fine line between infrastructure that simplifies your work and infrastructure that becomes a burden. When every new data pipeline requires juggling multiple tools for even basic tasks, it’s a sign that the balance has tipped too far. It’s one of the reasons I wrote the article Why Your Data Stack Won't Last - And How To Build Data Infrastructure That Will. We all want to build custom solutions or use new tools.
But then we are having a, can't see the forest for the trees, moment. We are just building infrastructure for infrastructure sake.
2) No Data Modeling is a Form Of Data Modeling
With so many tools making it easier to put data models into production, the speed at which we churn out messy, spaghetti-like data pipelines has only increased. Along with tables and data sets with far more limited usability.
As
pointed out, “no data modeling is a state of data modeling.” He even coined the term “just-in-time data modeling” (or query-driven data modeling). The point is clear: if you don't deliberately design a data model, a data model will be picked for you, and you'll likely end up stuck with something far more restrictive than intended.Sure, developing multiple SQL scripts and dashboards quickly might feel productive, but it comes at a cost. Trust me, I recall at my first job I developed a dozen dashboards relatively quickly with out a core data model. Then as people wanted changes or new dashboards I found myself having to do far more work than if I had built a core data model.
Eventually, someone will have to address the mounting tech debt, and it might far exceed what the team has the bandwidth to handle.
3) Build Things That Align With Key Business Metrics and Goals
A few weeks back I wrote about a business key metric and how to track it effectively.
It can be tempting to get caught up in technically complex projects that seem exciting on the surface. But if they don't drive a key business metric or align with a company goal, they’re unlikely to gain much recognition, no matter how clever or intricate the implementation is.
To build impactful solutions, you need to understand the core metrics that move the business forward. Consider the broader context: the macro environment, current trends, and internal initiatives. Even projects that seem straightforward can solve meaningful problems and provide real value when aligned with these goals.
4) Communication is a Super Power
Communication is one of the most valuable skills in any career, especially in technical roles. But often, communication, the skill gets abstracted away. People tell you to get good at it. Without telling you that there are so many skills inside of communication itself.
Here are just a few.
Whether it’s translating ideas into actionable steps for other developers or securing buy-in from the business, the ability to communicate at the right level is a superpower. Taking it a step further, the ability to craft a compelling story, especially when trying to get buy-in, can elevate your communication to the next level. There is communicating to get ideas across, then communicating to get people excited.
In particular, story telling was a skill I didn’t realize I should put much effort into until later on in my career. I assumed just talking to people was enough. Getting buy-in requires a lot more though. If you can excite people about an idea, and keep that excitement, you’ll be able to land far larger projects.
So if you have the chance to lead a project or deliver a presentation, do it. These opportunities are excellent ways to refine your communication skills and build confidence in sharing your vision.
5) Take Ownership Of What You Can
At the end of the day, your career or business is ultimately in your hands. While you can't control everything, there are key areas where you have influence:
When you say yes or no: Be intentional about the opportunities you choose to pursue. Early on in your career you’ll likely say yes to a lot because you need to learn, but as you go on, you’ll want to be more particular.
Seeking growth: Ask your manager or company for the chance to learn new skills or take on new projects. They might say no, but they might also say yes!
How you spend your time: Invest in activities that contribute to your growth, whether that’s learning new skills or tackling a personal project.
Small, consistent actions, like seeking feedback, building relationships, or dedicating a few extra hours to skill development, can lead to big changes over time. Yes, luck plays a role, but promotions and success aren’t handed out. Take ownership of what you can and put yourself in the best possible position for growth.
6) Solid Fundamentals Will Take You Far
Focusing on fundamentals might not be glamorous, but it’s essential for long-term success.
After all, every second spent learning SQL joins or understanding how to interact with an API might feel like time taken away from an opportunity to work with an LLM. With new technologies and best practices constantly emerging, it’s easy to feel like you’re falling behind.
However, a strong foundation of fundamentals enables you to adapt quickly and make sense of where new ideas fit. Instead of feeling left behind, you’ll find yourself equipped to integrate and leverage these advancements effectively.
7) Always Be Curious and Always Dig Deeper
When tackling a new problem or using a new technology, curiosity is your greatest asset.
Digging deeper not only helps you solve immediate challenges but also broadens your perspective, enabling you to see the bigger picture.
The best engineers and stakeholders I’ve worked with never settle for just knowing enough to get the job done. Instead, they could always dive deep into a subject, whether it was the origin of a specific design paradigm and its use cases or explaining how data is stored and processed across databases. Their curiosity allowed them to master the details while connecting those insights to broader systems and strategies.
What Lessons Have You Learned?
There are always more lessons to learn, and even relearn, throughout your career. Sometimes, it’s worth pausing to reflect and ask yourself:
What has happened in the past year?
Have any of your paradigms or beliefs shifted?
Are there recurring themes in what you’ve learned this year?
Has someone shared the same piece of advice with you multiple times recently?
It’s easy to feel like there’s no time for reflection with so much work to do. But without stepping back, we risk staying stuck in old habits or missing opportunities to embrace new, better ways of thinking and working.
Now that I’ve shared some of my lessons, I’d love to hear yours. What have you learned over the past few years that you believe others could benefit from? Feel free to comment below!
And as always, thanks for reading.
Video Of The Week: Building A Data Engineering Project - Starting A Data Analytics Project With "Real" Healthcare Data
Join My Data Engineering And Data Science Discord
If you’re looking to talk more about data engineering, data science, breaking into your first job, and finding other like minded data specialists. Then you should join the Seattle Data Guy discord!
We are now over 8000 members!
Join My Technical Consultants Community
If you’re a data consultant or considering becoming one then you should join the Technical Freelancer Community! We have over 700 members!
You’ll find plenty of free resources you can access to expedite your journey as a technical consultant as well as be able to talk to other consultants about questions you may have!
Articles Worth Reading
There are 20,000 new articles posted on Medium daily and that’s just Medium! I have spent a lot of time sifting through some of these articles as well as TechCrunch and companies tech blog and wanted to share some of my favorites!
How to give a senior leader feedback (without getting fired)
By
I got this question from a student in my October course cohort:
I’d love to hear any advice on helping people more senior than me recognize patterns of behavior that are stopping them from getting the results they want.
A few examples:
Not giving clear guidance during a process--then being disappointed with the results
Not setting priorities--then being disappointed their team isn’t working on the right things
Not training new hires--then being disappointed when they take longer to ramp up
Ah, this is a dilemma.
The Art of Execution: Making Things Happen as a Data Leader
If you’ve just been put in charge of a data team, you’ll be given some time to understand the lay of the land.
However, eventually, you need to start to make things happen.
And when it comes to leading data teams, progress doesn’t just happen. Usable data insights in whatever format doesn’t just happen.
Don’t get me wrong, you can spend weeks if not months doing tasks. However, no progress or impact can be made.
So when I say making things happen, I mean driving meaningful change in the business that benefits the business.
End Of Day 152
Thanks for checking out our community. We put out 3-4 Newsletters a week discussing data, tech, and start-ups.
Brilliant advice (and solid.writing too). Thanks!
Like the lessons learned.
Can you share what solid fundamentals to focus? (Point 6)