Centralized vs Decentralized vs Federated Data Teams
Setting Up Your Organization For Success
An important factor in succeeding when working in the data field is how you set up your team.
Now in my past articles, I have talked about onboarding and standards for data teams. However, I haven’t covered how data teams are organized.
Generally speaking, three key ways are described as how you can set up your data teams. But truthfully, many companies I have worked with will have a hybrid set-up of said three.
The general breakdowns are:
Centralized
Decentralized
Federated
Now, yes, perhaps team dynamics and organization might not be as exciting as discussing how the newest technology will change how we work.
But, ensuring your teams are organized efficiently is an important step in driving business impact. It impacts how effectively the business and data teams work together, and communicate, how different data teams inside the organization share information, and so much more.
Thus, in this article, I wanted to take a closer look into the different ways you can set up data teams and the implications each choice has.
Addressing Hybrid Teams
Before diving too deep, I do think it's important to state that data teams can actually use a mixed approach in terms of whether they are centralized, decentralized, or federated.
This is because you could have your entire data engineering team centralized while having analysts decentralized, or your entire data organization could be one big team serving the business.
So, as I write each section, realize that there can be plenty of mixes and matches regarding how the data teams are deployed.
Centralized Data Teams
Centralizing your data team is usually the first and most natural approach.
You can have your data engineers and analysts work together and have the business units reach out to them with their project needs. I feel as if many people believe this approach to data teams is mostly seen at start-ups.
However, I have worked with several large organizations that utilized a centralized data team approach. Generally, this was because the organization treated data more like traditional IT or viewed it as a cost more than a strategic partner.
In other words, they really wanted traditional reporting and not much more.
Pros of a Centralized Data Team:
Consistency: Having a centralized team ensures consistency in data management practices, tools, and technologies used across the organization. What tends to happen as companies expand is they pick up more and more solutions. Suddenly instead of just using Snowflake, a company might be using Snowflake, BigQuery, Postgres, etc. But when the data work is centralized you can avoid this(a little).
Economies of Scale And Limiting Duplicative Work: Centralizing resources can lead to cost savings as it eliminates duplication of efforts and allows for better utilization of resources. This actually wasn’t uncommon at Facebook. You might be taking on a project that another team or individual was working on and it’s hard to know if you are in your own little central data pod.
Knowledge Sharing: Centralizing the data team can facilitate better knowledge sharing and collaboration among team members. Whether its because they are physically in the same place and can converse daily on what is occurring in the business or just because they can all see the Kanban board.
Cons of a Centralized Data Team:
Slower Response Time: A centralized team may be slower to respond to individual business units' specific needs and requirements, as there may be more layers of bureaucracy and decision-making. This also can force your operations teams to simply create their own shadow data teams in many ways, defeating the whole purpose of the centralized approach. This is often amplified by smaller data teams who may not have the budget of a sales or marketing organization.
Risk of Being Out of Touch: The centralized data team may not have as deep an understanding of the specific needs and challenges of individual business units, leading to solutions that may not be as tailored or effective. It can also create an us vs. them feeling because instead of being aligned with the business your goals might be more focused internally.
Potential for Lower Engagement: Business units may feel less ownership and engagement in data projects if they are entirely managed by a separate centralized team. Again, this reiterates the Us vs. Them feeling. If the business doesn’t feel like the data team is being empathic or isn’t producing the results they want, they will seek out other options.
Why Companies Move Away From A Centralized Data Team
There are a lot of reasons companies will move away from centralized data teams. But here are a few reasons I have seen.
Marketing, Sales and other departments are building data teams on their own anyway as shadow data teams
Scale and rapid growth often pushes companies to start to decentralize their data teams so they can respond more effectively to business questions. Or at the very least they may just provide teams like sales with designated analysts.
A company may be looking to better align what their data analysts and other data professionals are working on and the individual business units goals.
Decentralized Data Teams
Decentralized teams are sometimes also referred to as embedded teams. Now, this is where things can get tricky; some companies I have worked with (or for) have a centralized data infrastructure and data engineering team.
However, the analysts are the ones that are decentralized or “embedded.” This provides a specialized analyst to operate internally and be better aligned with the department.
At the same time, the data engineering and infrastructure teams can benefit from the economies of scale by standardizing the core infrastructure(this actually can start to lead towards the federated approach will be referenced later).
Another approach could be integrating the data engineering and analysts into functional pods. But in my experience, this is when companies start to federate technology too.
Pros of Decentralized Data Teams:
Responsiveness: Decentralized teams can respond quickly to the specific needs and requirements of individual business units as they are closer to the action and have a deeper understanding of the particular challenges that the unit faces. Of course, this can be for better or worse. Suddenly a data analyst can become the “can you pull this data real quick” person. If there is no barrier between the analyst and the marketing team, then they might find themselves as a service org of one.
Tailored Solutions: Decentralized teams can develop solutions tailored to the specific needs and challenges of individual business units. Which can work for a while, but in my experience, having some form of central data repository, is beneficial. Even if everyone manages their own namespace or schema, it makes dealing with regulations and compliance(amongst other things) far easier.
Contextual Understanding: Probably one of the biggest benefits is the ability for data professionals to have a far better context on what they are working on. When you’re actually embedded into the team, you’re meeting with their leads and working on their problems daily, It can make it easier to figure out how to prioritize projects as well as deliver them.
Cons of Decentralized Data Teams:
Inconsistency: Having decentralized teams can lead to inconsistencies in data management practices, tools, and technologies used across different business units. This can lead to challenges with integrating data across the organization and may result in lower data quality and reliability. This also makes it harder to switch teams as new team members need to learn both the business context and the new tools.
Duplication of Efforts: Decentralized teams may lead to duplication of efforts as different business units may develop similar solutions independently. This can happen a lot, in fact when I worked at a hospital I worked on several dashboards that other teams were also working to deliver and we didn’t learn about the duplicative work until we both finished.
Challenges in Knowledge Sharing: Decentralized teams may face challenges in sharing knowledge and best practices across the organization as they are more siloed. That being said, I do find that when you have systems that encourage searching and looking through other people's code and work like data catalogs and automated wiki pages, this can be mitigated.
Difficulty in Aligning with Organizational Goals: I think this is likely the most salient point I felt when I worked for decentralized teams. Decentralized teams are very aligned with their business units but it’s easy to lose touch with the bigger picture.
If your team is looking for help improving your data team’s strategy or infrastructure, then set up a free consultation today!
Why Companies Look Into Adding A Federation Layer
I haven’t been in a situation where a company has, as a business goal, switched from a decentralized approach to a more federated approach. However, here are the reasons I assume it happens.
The company is dealing with far too many contracts and vendors and wants to simplify what they manage
The company wants to make it easier for its analysts, data engineers, and data scientists to shift between teams
A company may also want to re-center some of the data team's goals so they don’t just align with the business units but the organization as a whole.
Federated Data Teams
When I researched this topic, several articles referenced federated as decentralized and not a separate concept. But perhaps this is more because having a decentralized team can quickly lead to some federation of said teams and standards.
Federated, from my perspective, references the fact that the teams, the completed work, and the goals that the teams take on remain decentralized. The individual data teams are united under one company and its goals but their work is separated.
However, in terms of tooling and processes, those tend to be set by managed by a centralized team.
For example, at Facebook, most teams took on the work they felt was important to complete. However, when it came to tooling, there wasn’t much in terms of choice(although people were open to use what they wanted, the amount of support provided for the core tools made it difficult not to use).
And other teams managed and decided upon the core tooling and general infrastructure.
Everyone used Dataswarm for task orchestration.
Everyone used Spark or Presto to process data (and even that was through data swarm).
Everyone used Tableau or an internal BI tool for dashboards.
All those tables were centralized into a single data catalog and data lineage solution.
This made it easy for any data engineer to move between teams and allowed Facebook to streamline their onboarding. In addition, even if Facebook purchased solutions from external vendors, they’d likely benefit from reducing operations and maintenance contracts by picking single solutions like Presto or Dataswarm.
Pros of the Federated Model:
Contextual Understanding: Similar to the decentralized approach, since all the various teams still have a data team embedded, they will continue to maintain the context of what their business partners need.
Skill Development And Transferability: If all the teams are using the same solutions, then having data engineers or analysts transfer teams don’t have the same learning curve. The individuals only need to learn about the business and not the tools they are working with.
Economies of Scale And Duplicative Work: By centralizing the expertise and resources, the organization can achieve economies of scale and avoid duplication of efforts. You don’t need to have one team using Airflow, one team using SSIS, and another using who knows what. Instead, you can have one managed instance that everyone is using.
Cons of the Federated Model:
Potential for Bureaucracy: There is a risk that the federated approach could become bureaucratic and slow to respond to the specific needs and challenges of individual business units. This in turn can lead to a slowdown in innovation.
Slows Innovation: So this point is interesting. If you have a data infrastructure team that is very focused on constantly innovating and improving then you won’t deal with this issue. But if the core data infrastructure team isn’t incentivized to improve their core infrastructure, then when the individual business units need improved infrastructure, it may not happen.
Some Engineers May Find This Boring: At Facebook one of the issues that many data engineers faced(I know I had at least a dozen calls or 1:1s with them) was boredom. They thought when they were hired to work at Facebook they’d get to set up data infrastructure, work on Big Data problems, and write a lot more code. Of course, then they found that most of their work was very Airflow(like) and SQL heavy. Don’t get me wrong, some data engineering teams were very code and infrastructure-heavy. However because much of the baseline infrastructure was managed by software engineers, it took away some of the fun for some data engineers.
As a final thought about federated teams, I’d love to hear about a case where a company switched from a federated approach to a centralized approach. If that has happened.
Finding The Right Fit
Finding a data team structure that works and allows both the data teams not to feel overworked and out of alignment, as well as provide the business teams with results quickly, can help ensure your business has a successful data program.
Data teams can drive a lot of value for companies. However, it's crucial to put data teams in environments that allow them to provide said value. Yes, tools play a role in that, as do the individuals hired.
And so does how you organize your teams. The right team organization balances business alignment, speed, collaboration, and maintenance costs. These are key organizational considerations that need to be made.
Thanks so much for reading, and I will see you in the next one!
Data Events
Sign Up For Data Engineering Things And SDG Data Conference - Late October - Virtual
Data Vault User Group: Business Modeling to Data Vault Modeling - Virtual
Articles Worth Reading
There are 20,000 new articles posted on Medium daily and that’s just Medium! I have spent a lot of time sifting through some of these articles as well as TechCrunch and companies tech blog and wanted to share some of my favorites!
Dynamic Executor Core Resizing in Spark
Apache Spark™ is an open-source, distributed processing system used for big data workloads. It utilizes in-memory caching, and optimized query execution for fast analytic queries against data of any size.
Figure 1 depicts the spark cluster overview. The driver program is where most of the application logic runs alongside Spark’s scheduler and other infrastructure components, including those that talk to the Resource manager and get the necessary containers. The other is the executor, which gets work from the driver, executes it, and returns the result to the driver. In a cluster, there is only one driver and multiple executor processes, which might be short-lived compared to the driver’s run time.
End Of Day 94
Thanks for checking out our community. We put out 3-4 Newsletters a week discussing data, tech, and start-ups.
Excellent explanation and the visuals for each were interesting!
Wow great article 👏