You can post this every year and will be spot on every time 😅 I think that with new hype driven trends like databricks vs Snowflake, dbt vs SQLMesh, AI and the MDS speech we are always finding an excuse to avoid working on the right things, which are listed here and they will rock any other shinny project that involves AI or similar
I agree with all five points, and generally I’d add that if you’re an actual data leader get good at ignoring hype cycles. Not that you shouldn’t examine new tech, but most of it isn’t going to help you that much and is often just putting lipstick on the metaphorical pig of existing solutions.
Ten years on, the real enemies are still nouns, nulls, and n+1 dashboards. Model like you mean it; guard quality (AI will happily autocomplete your mistakes); pre-aggregate/cluster so dashboards don’t grow beards; keep batch by default—stream when someone’s pager says so. Most important: ship outcomes, not architectures. Tools are costumes; alignment is the plot.
Really enjoyed this! It’s interesting to see how, despite all the new tools, platforms, and AI capabilities, the core challenges in data engineering, like data quality, schema design, query performance, and aligning with business priorities, remain largely the same. Your point about business alignment is so true: technology alone isn’t enough if it doesn’t deliver real value. A great reminder that strong fundamentals still matter most.
LOL at the meme (soooo true) and a note on #3, definitely an art that needs to be applied thoughtfully as you said. Thanks to suggestions by LLMs I've seen coders indexing everything as a first pass on troubleshooting with no performance improvements... just more clutter to the database
You can post this every year and will be spot on every time 😅 I think that with new hype driven trends like databricks vs Snowflake, dbt vs SQLMesh, AI and the MDS speech we are always finding an excuse to avoid working on the right things, which are listed here and they will rock any other shinny project that involves AI or similar
I agree with all five points, and generally I’d add that if you’re an actual data leader get good at ignoring hype cycles. Not that you shouldn’t examine new tech, but most of it isn’t going to help you that much and is often just putting lipstick on the metaphorical pig of existing solutions.
Ten years on, the real enemies are still nouns, nulls, and n+1 dashboards. Model like you mean it; guard quality (AI will happily autocomplete your mistakes); pre-aggregate/cluster so dashboards don’t grow beards; keep batch by default—stream when someone’s pager says so. Most important: ship outcomes, not architectures. Tools are costumes; alignment is the plot.
Really enjoyed this! It’s interesting to see how, despite all the new tools, platforms, and AI capabilities, the core challenges in data engineering, like data quality, schema design, query performance, and aligning with business priorities, remain largely the same. Your point about business alignment is so true: technology alone isn’t enough if it doesn’t deliver real value. A great reminder that strong fundamentals still matter most.
LOL at the meme (soooo true) and a note on #3, definitely an art that needs to be applied thoughtfully as you said. Thanks to suggestions by LLMs I've seen coders indexing everything as a first pass on troubleshooting with no performance improvements... just more clutter to the database