Python and Go is not going anywhere. Java and JVM is here to stay.
However Rust and WASM are going to deliver backends that would outperform most of what exists today.
We know for sure that Rust would outperform the JVM based tools with ease and it would not only be faster, it would also be cheaper.
The biggest hurdle right now is the simplicity of implementation and integration, and the ease of use at the level of abstraction that data engineers are comfortable.
For me Rust scratches an itch to do some low level code, it's been a long time since I've been able to do that. With DE getting so platformized (DataBricks, Snowflake, DBT) this job is becoming more about slinging YAML and HCL.
I think if you were writing Redis or Kafka today, Rust would be a strong choice, but it's not going to replace Python. The overall friction is way too high. I like that it makes me think about ownership and error handling up front, but that definitely makes for slower development and I don't see the data science cohort becoming Rust enthusiasts. On the other hand, I just don't see a reason to use C/C++ anymore, Rust seems like a much better choice for those applications where you'd reach for C. Java is meh, there's a lot of infra and it's the easiest path for existing integrations, but Java itself is just a bad language. The JVM alternatives like Scala and Kotlin have nice features and are nice languages but in the end you're still on the JVM with its attendant headaches. I observed to a friend recently that we've traded time saved chasing resource leaks with time wasted dealing with the operational headaches of GC.
Generally agree with the overall thought process.
Quick question:
Have you come across Fluvio - https://github.com/infinyon/fluvio ?
Python and Go is not going anywhere. Java and JVM is here to stay.
However Rust and WASM are going to deliver backends that would outperform most of what exists today.
We know for sure that Rust would outperform the JVM based tools with ease and it would not only be faster, it would also be cheaper.
The biggest hurdle right now is the simplicity of implementation and integration, and the ease of use at the level of abstraction that data engineers are comfortable.
For me Rust scratches an itch to do some low level code, it's been a long time since I've been able to do that. With DE getting so platformized (DataBricks, Snowflake, DBT) this job is becoming more about slinging YAML and HCL.
I think if you were writing Redis or Kafka today, Rust would be a strong choice, but it's not going to replace Python. The overall friction is way too high. I like that it makes me think about ownership and error handling up front, but that definitely makes for slower development and I don't see the data science cohort becoming Rust enthusiasts. On the other hand, I just don't see a reason to use C/C++ anymore, Rust seems like a much better choice for those applications where you'd reach for C. Java is meh, there's a lot of infra and it's the easiest path for existing integrations, but Java itself is just a bad language. The JVM alternatives like Scala and Kotlin have nice features and are nice languages but in the end you're still on the JVM with its attendant headaches. I observed to a friend recently that we've traded time saved chasing resource leaks with time wasted dealing with the operational headaches of GC.