Great follow-up to Part 1! 👏 I really appreciate the deep dive into how the data engineering landscape is shifting—especially the points around the increasing convergence of data engineering and analytics. The growing focus on real-time data and modular architectures definitely resonates with what we’re seeing on the ground too. It's clear that the role of a data engineer is evolving from just pipeline builders to strategic enablers of business intelligence. Curious to hear your thoughts on how low-code/no-code tools will impact the field going forward. Keep these insights coming!
This sentence did it for me: "There is always talk about garbage in garbage out being a major issue, especially when you not only want to build basic analytics but also deploy machine learning models." I haven't given too much thought on Data Catalogs; but I will now. Awesome stuff!
Not recalling if you did a breakdown of the 400 respondents and the percentage of those across size of company in general, but seeing such 'low' counts for more 'legacy' integration tools like Informatica, Talend, Matillion is surprising, as well as not seeing mention of integration tools like Airflow, Workato, Mulesoft, Boomi, SnapLogic, etc.
Do you think that the data folks that follow/interact with you are more geared towards MDS and not the 'data architects' of the pre-MDS rise?
Great follow-up to Part 1! 👏 I really appreciate the deep dive into how the data engineering landscape is shifting—especially the points around the increasing convergence of data engineering and analytics. The growing focus on real-time data and modular architectures definitely resonates with what we’re seeing on the ground too. It's clear that the role of a data engineer is evolving from just pipeline builders to strategic enablers of business intelligence. Curious to hear your thoughts on how low-code/no-code tools will impact the field going forward. Keep these insights coming!
https://kaliper.io/data-engineering-services/
Hey Ben!
Thank you for your investigation!
I would love to see the databricks and snowflake adoption as well as cloud competition: aws vs gcp vs azure.
My feeling is GCP is catching up very aggressively.
This sentence did it for me: "There is always talk about garbage in garbage out being a major issue, especially when you not only want to build basic analytics but also deploy machine learning models." I haven't given too much thought on Data Catalogs; but I will now. Awesome stuff!
Great to SSIS is still relevant and it’s cloud counterpart Azure Data Factory. Good article.
Not recalling if you did a breakdown of the 400 respondents and the percentage of those across size of company in general, but seeing such 'low' counts for more 'legacy' integration tools like Informatica, Talend, Matillion is surprising, as well as not seeing mention of integration tools like Airflow, Workato, Mulesoft, Boomi, SnapLogic, etc.
Do you think that the data folks that follow/interact with you are more geared towards MDS and not the 'data architects' of the pre-MDS rise?