I have to come clean — I build data pipelines, but I’m not a Data Engineer; at least according to some top tech companies. Over the past few years at one Silicon Valley company, I’ve worked with many Data Engineers. To be more specific; varying roles who deliver work commonly aligned with data engineering, such as writing code to build data pipelines. We’re in a time period where data engineering doesn’t have a widely agreed upon definition. While some companies, such as Amazon and Meta lean on Data Engineers to define metrics, write SQL, and even build dashboards, other companies task Data Engineers with coding entire applications in languages like C++ or Java.
Based on jobs listed on metacareers.com, Data Engineers in Meta’s own terminology are typically an extension of software engineering. It is common to see listings for “Software Engineer — Data Engineering”, and less common to see titles such as “Data Engineer” or “Business Intelligence Engineer.” Meta is one of the most mature engineering cultures founded on engineering principles, and engineering can have a very specific meaning.
The bar for Data Engineers at Meta is just as high — and sometimes higher, than that of an equivalent Software Engineer at Meta
How could it be bad for Data Engineers with a company like Meta on their resume? Firstly, according to many Glassdoor reviews, the bar for Data Engineers at Meta can be just as high — and sometimes higher, than that of an equivalent Software Engineer at Meta. At that point, you’ve strayed so far from the industry definition of Data Engineer, and have garnered a salary inline with that of a Software Engineer, that it wouldn’t then make sense to continue to other organizations performing Data Engineering duties. One would then be taking quite a demotion, and at that point, it may make more sense to continue forward with software engineering.
Who is building data pipelines? Most often; Software Engineers, but increasingly, business intelligence analysts and other analytic-focused roles. These roles commonly accept significantly lower salaries than that of Business Intelligence Engineers or Data Engineers at other leading technology companies. I believe that this gap in data infrastructure causes two problems. Firstly, companies aren’t valuing the roles that are…