Member-only story
Data Engineering Career Path: To Code or to No-Code?
Which direction do I have to grow in before ultimately topping out in this discipline?
As a Sr. Data Engineer, I find myself questioning what’s next. Many companies consider their Data Engineers an extension of analytics (I’m looking at you, Meta.) Many others, such as Apple and Netflix, see their Data Engineers as an extension of Software Engineering. My current skills are more focused towards low-code and no-code, such as writing SQL and scheduling scripts in tools such as DBT/Airflow. While I have consumed APIs, I certainly haven’t written any. I began to question; where do low-code/no-code Data Engineers go from here? Is there a ceiling in sight? Which direction do I have to grow in before ultimately topping out in this discipline?
I believe this divide has been caused by the excellent data platform products from the likes of DBT, Snowflake, DataDog, Palantir (if anyone actually knows what they do, hit me up) and DataBricks. These tools allow SQL-specialists to harness the transportation capabilities relegated to the Software Engineers or Data Engineers of times past. Clearly these tools are empowering companies to deliver value at a higher velocity, and the market is taking notice.
Why Do Data Engineers Code?
In organizations that see Data Engineering as an extension of Software Engineering, Data Engineers wear many hats. Building APIs, designing data models, designing payload structures, and building data platform infrastructure can all fall under Data Engineering. The most frequent languages (after Python) appear to be Java, Scala, and C++. These Data Engineers often work with streaming technologies such as Kafka and Flink, as well as DevOps infrastructure like Docker, Kubernetes, and Glue. These Data Engineers may often partner with Software Engineers; the Software Engineers can focus on building application functionality, and Data Engineers can focus on extracting value from the application data.
Pros:
- Able to solve a broader scope of engineering challenges
- Often paid higher on average
Cons:
- Can be over-skilled and under-utilized
