Let's Move Fast And Get Rid Of Data Engineers

We live in a data world that wants to move fast.

Just shove your data into a data lake and we’ll figure it out later.

Create this table for this one dashboard that we will only look at once and then it will join the hundreds(if not thousands) of other dashboards that are ignored.

All of which can work in the short term but often leaves behind a lot of technical debt. One project I recently worked on literally dumped all of their raw data into an S3 bucket with no form of breakdown source or timing and it was quite chaotic to grasp what was going on.

All of this is driven by executives who need their data yesterday and don’t want to wait as well as by the software and analysts teams who are often driven by very different motivations.

With all this pressure to move fast coming from all sides, an interesting solution I have come across a few times is…Let’s just get rid of data engineering and governance.

Let’s Just Get Rid Of Data Engineers?

Inspired By Codestrap