Data Scientists in Software Teams: State of the Art and Challenges – TSE 2018

by Miryung Kim, Thomas Zimmermann, Robert DeLine, Andrew Begel

The demand for analyzing large scale telemetry, machine, and quality data is rapidly increasing in software industry. Data scientists are becoming popular within software teams, e.g., Facebook, LinkedIn and Microsoft are creating a new career path for data scientists. In this paper, we present a large-scale survey with 793 professional data scientists at Microsoft to understand their educational background, problem topics that they work on, tool usages, and activities. We cluster these data scientists based on the time spent for various activities and identify 9 distinct clusters of data scientists, and their corresponding characteristics. We also discuss the challenges that they face and the best practices they share with other data scientists. Our study finds several trends about data scientists in the software engineering context at Microsoft, and should inform managers on how to leverage data science capability effectively within their teams.

Download as PDF.


Miryung Kim, Thomas Zimmermann, Robert DeLine, Andrew Begel. Data Scientists in Software Teams: State of the Art and Challenges. In IEEE Transactions on Software Engineering: (2018), 2018. To appear

BibTeX Entry

    title = "Data Scientists in Software Teams: State of the Art and Challenges",
    author = "Miryung Kim and Thomas Zimmermann and Robert DeLine and Andrew Begel",
    year = "2018",
    journal = "IEEE Transactions on Software Engineering",