Why Data Engineering Has Overtaken Data Science

It was October 2012, Data Science seemingly exploded overnight with the publishing of the now-famous Harvard Business Review article titled “Data Scientist: The Sexiest Job of the 21st Century”. This triggered a tidal wave of interest in advanced analytics, was the impetus for many graduate Data Science programs, and careers exploded. While model building is often touted as the sexy part of the Data Scientists role, the truth is that Data Scientists are expected to wear many hats. To produce value, the vast majority of a Data Scientist’s time is spent meeting frequently with business stakeholders, writing mountains of SQL to clean dirty data, and even more SQL when building features for their models. I believe Data Science is driven by misconceptions about the nature of the work.

Since the beginning of popularity of Data Science, one critical oversight has crippled Data Science teams. Data Scientists have been spending a great majority of their time performing duties outside of building and tuning models; often appearing quite similar to their Analyst and Business Intelligence Engineer counterparts. Before rewards can be reaped from predictive models, the data must be created by software applications, transported to data warehouses, and cleaned. These early Data Scientists ultimately had to adapt and become generalists in many disciplines before specializing…