Output details

A systematic review of systematic review process in software engineering.

<33> Research has identified a continuing increase in the adoption of systematic reviews (SRs) in software engineering (SE). Most studies have used the Kitchenham guidelines for SRs in SE which were based on medical and social science experience. To keep the guidelines up to date, this SR identified, evaluated and synthesized studies published by SE researchers concerning their experiences of performing SRs and their proposals for improving the process. Results will form the basis of an up-date to the SR guidelines that will be part of a book on evidence-based practice in SE that Chapman & Hall have agreed to publish.

This paper provides new knowledge about the process of applying the systematic review (SR) method within the software engineering (SE) domain. It is a comprehensive study using a systematic approach to aggregate outcomes from the 63 unique studies found. Although, the study indicated that the current guidelines have been well-received, there are several practical problems:

• Digital libraries in SE are not well-suited to complex automated searches

• Substantial time and effort are needed to carry out SRs

• Assessing study quality is difficult particularly where papers are based on a range of different research methods

The paper presents evidence to support a set of recommendations for changes to the SR guidelines. In particular in contrast to medical and sociological SR guidelines, in software engineering:

• The use of structured questions to construct complex Boolean search strings is inappropriate.

• Quality evaluation criteria are needed to cater for technology intensive experiments.

• The use of a quasi-gold standard set of known relevant papers can provide a good means of integrating manual and automated searches and qualify the completeness of search strategies.

Furthermore, the quasi-gold standard approach is likely to be of use to researchers in other disciplines.