What It Really Takes to Move From Senior to Staff Data Engineer
An Interview With A Staff Level Data Engineer At Apple
Hi, fellow future and current Data Leaders; Ben here 👋
Today’s article is an interview with Brian Femiano who is currently a staff data engineer at Apple.
He’s been doing data engineer for nearly 20 years with 9 of those as a staff eng. He’s worked across many domains including ad tech, music, social media, and intelligence. In his own words he enjoys mentoring tech professionals and helping folks level up, especially his managers. He lives with his wife Rachel and two boys.
Now let’s jump into the article!
Once you start pushing past senior-level, whether that’s as a data engineer, analyst or other IC, there are a lot of skills that start becoming important that have nothing to do with technology.
Or at least less so.
I’ve already written several takes on this in the past so I wanted to bring in someone else’s perspective. So for this article we have Brian Femiano who is currently a staff data engineer at Apple(in case you missed the intro) and he’ll be sharing about:
Career Path And Motivation
Defining The Staff Role
Technical Design & Systems Thinking
Collaboration & Communication
Big-Picture Advice
If you’re stuck at senior or are just curious what it takes, then this article is for you!
Career Path & Motivation
Q. How did you get into data engineering? And what do you enjoy about it?
I started with Java services then one day got into the early Cloudera HFS/MapReduce training videos and got hooked. Distributed computing finally felt approachable for me and was really cool to see applied to data and not just number crunching.
Q. Can you share the key moments or projects that shaped your path from senior to staff data engineer?
Years ago I was on a team and someone needed to organize work into JIRA, assign that work out, document progress, help unblock teammates, and keep stakeholders outside of engineering informed. It’s not the kind of fun stuff devs enjoy doing but it had to get done. Once the org realized I had this holistic view of the project goals and how all the work could be organized to meet them, they recognized me as staff.
Defining the Staff Role
Q. How would you describe the difference between senior and staff data engineer in your own words?
It’s not an automatic flip so much as a gradual development. A senior dev might stick to a comfort zone and execute really well. The next step is non-coding skills that make other devs on their team better.
Things like:
Diagramming systems and runbooks to help on-call
Having good relationships with product folks
Having a good relationship with other data teams to know how your changes impacts them.
Staffs also have to context switch more frequently in a given day.
Note From The Seattle Data Guy - I’ve always enjoyed the different Calendars on Will Larson’s website where he discusses many of the different archetypes of staff engineers. As Brian referenced about context switching, you might have to go from mentoring a junior engineer, to talking about scaling issues, to then discussing budget, interviewing, planning offsites, etc. Actually, in the next answer Brian will reference his thoughts against the archetypes idea. Which I do understand. I could write another article about my thoughts there, but for now let’s go back to Brian!
Q. What misconceptions do people have about the staff title?
That you need to be the most talented developer on your team
That everyone fits into these nice little archetype roles like classes in an RPG video game.
That job hoping is the best way to get the title.
Q. What do you find is the most common blocker for senior data engineers trying to get to staff ?
Poor or non-existent communication or being abrasive. Not flexible based on circumstances or able to see the big picture of their work. Being too focused on tools/languages and just generally not building for long term system health and quality.
Q. For a senior engineer aspiring to staff, what specific signals should they show?
There’s a ton but I’ll rattle a few off.
That their managers and peers know they think holistically about the systems you’re building. Thoughts are put into diagrams and accept feedback well.
Support your teammates during fires.
Generally you want to make yourself indispensable to your team but not in a way where you have siloed knowledge or also always having to play the hero.
It’s a fine line to walk.
Technical Design & Systems Thinking
Q. When faced with a complex request, what’s your process for designing a simple, scalable solution?
Immediately diagram my thoughts.
What currently exists that might help satisfy the request?
Are we happy with those systems or is this our chance to refactor problem areas?
What’s the minimalist set of new capabilities needed to be built?
Can we leverage existing libraries?
Are any parts of the design susceptible to bottlenecks as load increases?
Does any part of this need constant manual attention or is it mostly automated?
Q. How do you decide what to build yourself versus what to delegate or break into
components?
This is a tough one. The team leads I learned from tried not to involve themselves on the critical path. I’ve broken this rule many times though based on circumstances. If you assign yourself too much it can delay the project and also deprive your teammates of growth, both of which hurts the business. Generally you want to trust the most important areas to your teammates. Figure out where you can pitch in to help avoid overload.
Q. Can you share an example of a “simple” solution to a complex problem that you’re
particularly proud of?
Back at Pandora we wanted a service that would notify artists when one of their songs was added to certain DJ curated playlists.
The business objective was clear: If we send them sharable links, then they might post them organically and give us free publicity.
We started out thinking we’d fire emails in real time for any new Kafka events, but this had a number of unattractive tradeoffs.
After some back and forth with product we realized artists didn’t even want the notifications in real time. We ending up building a minimal set of reliable components that read the events and batch the emails.
It’s still in prod today.
Collaboration & Communication
Q. What have you learned about keeping managers, product partners, and other engineers aligned on big projects?
With management I try to be succinct and timely so they can stay informed on progress with minimal effort. For product it’s a lot of asking questions and note taking. Repeating back things they tell me in my own words. Engineering it’s about giving as much detail and clarity, even if it means repeating yourself many times. Exercising patience with everyone. I’m still working on all
of this.
Q. How do you create an environment where teammates feel comfortable approaching you with questions or concerns?
I think it starts with just being friendly and non-judgmental so they can ask me anything. I try to prioritize anyone who’s asked for my help to unblock them. Also give your teammates the leverage when they speak up and have a good idea you didn’t think of.
Q. Any tips for writing design docs that earn trust and reduce confusion?
Start out with 3-5 sentences outlining what’s being built and what are the business benefits. Reference other parties involved in the project so readers know everyone involved and can reach out themselves. Find a diagramming tool you like and focus on diagrams that communicate how pieces fit together. It’s less about how good the diagrams look artistically and more about how easy they are to follow.
Big-Picture Advice
Q. Where do you see data engineering going over the next few years. Anything you’re most excited about?
We’re already starting to see data engineer roles looking for proficiency in languages beyond just Java/Scala/Python/SQL and that will continue. It’s also been cool to see orgs recognizing that their problems aren’t with volume but more governance and data quality, and choosing tools/platforms more aligned with that reality.
It’s exciting to see some of the open table formats evolve and help teams with those areas. I also have a somewhat optimistic take on GenAI in this space. There’s so much talk of it replacing entry level roles but I don’t think so. The entry level folks I interact with are the ones really good with it and schooling the seniors.
Note From Seattle Data Guy - If you’d like to learn more about Brian and his past work, you can also check out the video below!
Final Thoughts
This is Ben Rogojan again!
What a senior and staff engineer is means different things at different companies(actually I guess I could say this is even true amongst teams). I’ve seen some be more focused on architecture, others coding machines and still others play a mix of a technical project manager, strategic partner to the business and mentor to more junior engineer.
The through line is that these engineers have some form of outsized impact. We all only have so many hours in the day and we can only code so fast. At a certain point we have to start considering more about what we are coding vs. just the speed.
For example:
Finding and working on projects that move the business forward
Helping the business remove ambiguity
Keeping the business from picking a solution that will cost them millions and require them to migrate in 18 months.
I am so glad Brian shared his experiences. I hope you found them valuable!
Events
Articles Worth Reading
There are thousands of new articles posted daily all over the web! I have spent a lot of time sifting through some of these articles as well as TechCrunch and companies tech blog and wanted to share some of my favorites!
Beyond Big Tech: The Reality Of Data Engineering Outside Silicon Valley
I’ve had the privilege of working at a mix of companies—big tech, start-ups, enterprises, and nonprofits. Each has given me a unique perspective on how different environments approach data: the infrastructure they build to manage it and how they ultimately use it.
I’m writing this because, while tinkering on a small side project, I was hit with a vivid reminder of just how frustrating it can be to work with data from certain industries. In fact, when I made a humorous post about it, someone called out the poor data model in the raw file I was using.
But that’s the thing: it wasn’t my model, that’s just how the data came.
Building Zone Failure Resilience in Apache Pinot™ at Uber
Initially, our Pinot clusters at Uber relied on two key strategies: tag-based instance assignment, which groups servers by tenant to ensure logical isolation, and balanced segment assignment, which spreads data segments evenly across those servers. While effective for distributing data evenly across servers within a tenant, this approach didn’t inherently guarantee distribution across different physical zones. If all instances assigned to a table, or all replicas of a segment, happened to be in a single zone, a failure in that zone would lead to significant service disruption.
End Of Day 202
Thanks for checking out our community. We put out 4-5 Newsletters a month discussing data, tech, and start-ups.
If you enjoyed it, consider liking, sharing and helping this newsletter grow.
The context switching is one of the big ones for me. That and the full end-to-end ownership of a project. It’s not enough to just assist with any data pipelines. You have to have a comprehensive approach which might include infrastructure as code, business intelligence, topology diagrams, clear documentation, slide decks, and leading cross functional meetings. You don’t have to know everything, but you have to know a lot.
Thank you for sharing!