Please submit to the 1st International Workshop on Anti-Patterns for Software Analytics (APSA 2018), in conjunction with ICSE 2018, Gothenburg, Sweden. The submission deadline is February 5 — as always, please check the webpage for any extensions. I’m a member of the Program Committee.
In this data-driven economy, as society makes increasing use of data mining technology, it is now more important to that our community has a shared understanding on how to assess the results coming out of those data miners. Recent results experience shows that, in the arena of software analytics, we do not share that understanding.
We now have more than a decade of research on data mining in software repositories, reported at all major software engineering venues (ICSE, TSE, EMSE, MSR, ASE, ICSME, ESEM, …). Based on the organizers’ experience on their last dozen journal papers, we assert that conference and journal reviewers in SE have very little shared criteria on how to assess data miners. Simple low-level issues, such as what performance indicator to use, are still controversial. Some reviewers eschew accuracy or precision; some demand SE (standardized error). Similarly, many higher issues are also unclear such as what statistical test to use on how many data sets (and where should that data come from). More generally, recently several papers reported on failed replications or problems with the data we use.
All the above hints at general and systemic problems with the way we evaluate and compare our research. This is a pressing, open and urgent problem not just for researchers since we know many software developers who routinely ship some kind of analytics functionality as part of their delivery tools. If we, as academics, cannot agree on how to assess those tools, then how can industrial practitioners ever certify that the analytic tools they are shipping to clients are useful (or, at the very least, not misleading).
Accordingly, this workshop’s goal is the development of guidelines for assessing software analytics. We want to bring together the community to discuss anti-patterns as a first step towards guidelines for repeatable, comparable, and replicable software analytics research, e.g., on defect prediction and effort prediction. As such, we do not want to discuss new techniques, data sets, or ways to mine data, but instead focus solely on the discussion of how we should actually evaluate our research. This shall give researchers a forum to share anti-patterns they frequently observe and how to avoid them.