Are companies letting down data engineers, data scientists, and software engineers? It feels like some may be. Some companies aren’t creating environments where newly graduated technical practitioners can grow. Instead, in many cases, they are sometimes left to their own devices, given poor
Firstly, good work on this article, the flow of your points are well structured and your words are well written. Based on what you have described, you have clearly been fed to the wolves and put through the gauntlet. Do not let this discourage you or make you think your company is doing this on purpose to torment you as you do reverse engineering (again) on your next project and left to fend for yourself (again).
There is a data engineering joke of a kind of email you can get in your nightmares, “good day, you are being pulled in today to work on a critical data project, which is in firefighting mode because it was written all so poorly and nothing is loading, and while you’re at it, please document the system as well?”
Jokes aside, what you have gone through and continue to go through are birthing pains (another overused idiom in this industry) that will, in the course of a few more short years, forge in you the strong foundations of a senior data engineer …the one you never had.
In our line of work, no one is owed complete documentation, squeaky clean data, pipelines don’t have issues, and processes that do what they are supposed to do. Some projects are even to do exactly just that (there are projects and teams out there in the multi million dollar industry of documenting, reverse engineering, untangling, and optimizing spaghetti data platforms—another man’s trash is another man’s treasure indeed!)
But that doesn’t mean it has to be that way, at least not all the time.
That being said, if you are a junior data engineer and are reading this, the best advise I can give is to know what you do not know as much as knowing what you do know.
The best junior data engineers I have worked with were not always the smartest ones. It was the one who tried to set boundaries and expectations, raises their hand when asking for help, and says no. Don’t be afraid to say you can’t, but also be specific in what help you need. Then be eager to step up where you can.
There is zero talent needed for being dependable and showing up.
All the best!! And keep up the great work here. —Alfonso R.
A lot to unpack here, in general too many companies tend to focus on outcomes and not process which can contribute to the single person dependency issue that is so common for businesses of all sizes. As a non-technical founder, I'm 100% reliant on TRUST and HOPE that developers are following best practices what could go wrong? After learning first hand what can go wrong, short answer: everything. I now contract with a third party to audit developer processes for documenting all the work being done, with the goal that should I be able to hire a new developer, a new development firm and they should be able to pick-up where the previous team left off. It's not cheap, but it does help me sleep better at night. I may be ready to hire my first developer this year, I'll spend the money to hire a senior one. Thanks Ben!
Firstly, good work on this article, the flow of your points are well structured and your words are well written. Based on what you have described, you have clearly been fed to the wolves and put through the gauntlet. Do not let this discourage you or make you think your company is doing this on purpose to torment you as you do reverse engineering (again) on your next project and left to fend for yourself (again).
There is a data engineering joke of a kind of email you can get in your nightmares, “good day, you are being pulled in today to work on a critical data project, which is in firefighting mode because it was written all so poorly and nothing is loading, and while you’re at it, please document the system as well?”
Jokes aside, what you have gone through and continue to go through are birthing pains (another overused idiom in this industry) that will, in the course of a few more short years, forge in you the strong foundations of a senior data engineer …the one you never had.
In our line of work, no one is owed complete documentation, squeaky clean data, pipelines don’t have issues, and processes that do what they are supposed to do. Some projects are even to do exactly just that (there are projects and teams out there in the multi million dollar industry of documenting, reverse engineering, untangling, and optimizing spaghetti data platforms—another man’s trash is another man’s treasure indeed!)
But that doesn’t mean it has to be that way, at least not all the time.
That being said, if you are a junior data engineer and are reading this, the best advise I can give is to know what you do not know as much as knowing what you do know.
The best junior data engineers I have worked with were not always the smartest ones. It was the one who tried to set boundaries and expectations, raises their hand when asking for help, and says no. Don’t be afraid to say you can’t, but also be specific in what help you need. Then be eager to step up where you can.
There is zero talent needed for being dependable and showing up.
All the best!! And keep up the great work here. —Alfonso R.
https://www.linkedin.com/in/jose-alfonso-ramirez-a9686aa7
A lot to unpack here, in general too many companies tend to focus on outcomes and not process which can contribute to the single person dependency issue that is so common for businesses of all sizes. As a non-technical founder, I'm 100% reliant on TRUST and HOPE that developers are following best practices what could go wrong? After learning first hand what can go wrong, short answer: everything. I now contract with a third party to audit developer processes for documenting all the work being done, with the goal that should I be able to hire a new developer, a new development firm and they should be able to pick-up where the previous team left off. It's not cheap, but it does help me sleep better at night. I may be ready to hire my first developer this year, I'll spend the money to hire a senior one. Thanks Ben!