lol literally announced today, I considered putting a little blurb about it, like "Oh hey look never mind lets do ducklake instead, it'll be the real real savior"
Well, you can't finish the journeys that you don't start, so you're never going to arrive at the nirvana of centralised data if you don't start somewhere.
More and more major platforms are going down the direct connect route, for example, Salesforce Data Cloud, which is white-label Iceberg, bi-directional to Snowflake. There's a good chance that all major platforms will have to provide an Iceberg connector and we may have the chance of reaching this ideal world of a federated data mesh with an Iceberg front-end layer.
The best use case I've seen is bridging the gap between analysts/data scientists who want all the data regardless of a use case and the desire to manage your Snowflake costs. There is no reason to bring thousands of tables into Snowflake and update them when you can stage them in a query-able layer in S3 and port them into Snowflake with an external table when you have a use case for them.
iceberg is so last week #ducklake
lol literally announced today, I considered putting a little blurb about it, like "Oh hey look never mind lets do ducklake instead, it'll be the real real savior"
Cast off the shoes. Follow the gourd!
some days I wish I was still in my old ssis echo chamber
Logging into a windows VM waiting for visual studio to load!? 🤣
can't wait for it to load if your project is so big visual studio won't open so you're forced to use biml 😂😂😂
We chose this road, we must now continue down it haha
Well, you can't finish the journeys that you don't start, so you're never going to arrive at the nirvana of centralised data if you don't start somewhere.
More and more major platforms are going down the direct connect route, for example, Salesforce Data Cloud, which is white-label Iceberg, bi-directional to Snowflake. There's a good chance that all major platforms will have to provide an Iceberg connector and we may have the chance of reaching this ideal world of a federated data mesh with an Iceberg front-end layer.
The best use case I've seen is bridging the gap between analysts/data scientists who want all the data regardless of a use case and the desire to manage your Snowflake costs. There is no reason to bring thousands of tables into Snowflake and update them when you can stage them in a query-able layer in S3 and port them into Snowflake with an external table when you have a use case for them.
Historically, icebergs have never been a very dependable vehicle of rescue... :-)
I considered creating an AI generated image of an Iceberg coming to save the titanic
You get me! My thoughts exactly.
Iceberg is incredible, but it is also too low-level and has poor usability for small data sets.
Wrote blog post about it:
https://quesma.com/blog-detail/apache-iceberg-practical-limitations-2025