Speed Running The Data Infrastructure Industry

Over the past few years we have watched as start-ups in the crypto currency and data infrastructure space essentially speed ran their industries.

Repeating the same discoveries that data management organizations like DAMA and TDWI have codified a long time ago. The need for better security and governance of data becoming very apparent as cases such as the OpenSea and Customer.io fiasco become public(And thats just one of the dozens that we are actually hearing about).

It’s easy to see why experienced veterans of the data space feel that all the tools of the Modern Data Stack(MDS) are far from new and encourage a lack of rigor or standards.

Runaway cloud costs and the continuation of data swamp building from our data lake days haven’t left us.

It hasn’t been a loss though. I do believe there has been a benefit to the speed running of the data infrastructure industry. The large influx of cash and variation of tooling has challenged a lot of norms and in some cases taken advantage of new paradigms such as the cloud.

And it would be a waste if we didn’t take a moment to reflect how this time, it’s different.

Is It Different This Time

A line often stated in any period of irrational exuberance is that “this time it is different”.

It was different when Hadoop became standard and it was different as the popularity of the MDS grew.

Yet many companies still struggled to answer questions about their data. On top of that companies still struggle to manage costs and track runaway pipelines that are creating security risks.

But to say things haven’t changed would be missing the role that cloud has played for many companies. Especially those companies that are smaller and would have never even considered tracking their data in the past because it would have required a team of 3-4 people just to stand up server and manage their network.