Evidence-Based Software Engineering

EBSE for...

EBSE for Researchers

What is the Evidence-Based Paradigm?

The essence of the evidence-based paradigm is that of systematically collecting and analysing all of the available empirical data about a given phenomenon in order to obtain a much wider and more complete perspective than would be obtained from an individual study, not least because each study takes place within a particular context and involves a specific set of participants. While these factors can bias the outcomes from an individual study, taking a wider view should make it possible to produce more reliable conclusions and to minimise the effects of local factors.

The core tool of the evidence-based paradigm is the Systematic Literature Review (SLR). A systematic literature review (often abbreviated to systematic review) is an established research method that has been formulated to help with answering research questions in those domains where individual empirical studies are not easily controlled or replicated. It is a form of secondary study, aimed at gathering together and analysing all of the experimental results available for a topic in an objective, unbiased and consistent manner. (Within this context, a primary study is a single experimental study, which may take a range of forms including surveys, case studies, randomised experiments, quasi-experiments...)

Systematic reviews originated in clinical medicine, where two decades ago it was recognised that the outcomes from individual clinical trials were not a safe or sufficient basis for decision making. Although clinical medicine can make use of randomised controlled trials (RCTs), where techniques such as double blinding (neither the researcher nor the recipient know whether a 'treatment' or a placebo are being administered) help with objectivity, there are still many sources of possible bias within an individual study. Subsequently their use has spread into other domains where such experimental rigour is either difficult or impossible to achieve, and where many different experimental forms may well be employed.

(Click here for more information about Systematic Literature Reviews.)

A more 'open' form of review is provided by a mapping study (also termed a scoping review). This reviews a specific software engineering topic and classifies the primary research papers for that specific domain. The research questions for such a study are quite high-level, and include issues such as which sub-topics have been investigated, what empirical forms have been used, and which sub-topics have been addressed by enough studies to merit a more detailed systematic review. The table below shows the differences between the two forms, but the main distinction is that where an SLR seeks to aggregate the outcomes of primary studies, a mapping study aims only to classify literature and aggregate studies within the categories.

Review Process Mapping Study Systematic Literature Review
Goals Classification and thematic analysis of literature Identifying best practice
Research Question General -- related to research trends: which researchers, how much activity, what type of studies, etc. Specific -- related to the outcomes from empirical studies.
Of the form: "is technology / method A better or not than B"
Search Process Defined by topic area Defined by research question
Scope Broad -- all papers related to a topic area are included, but only classification data about those are collected Focused -- only empirical papers related to a specific research question are included and detailed information about individual research outcomes is extracted from each paper
Search strategy requirements Less stringent if only research trends are of interest Extremely stringent -- all relevant studies must be found
Quality evaluation Not essential Important to ensure that results are based upon best quality evidence
Results Set of papers related to a topic area, categorised in a variety of dimensions and counts of the number of papers in various categories Answer to specific research question, possibly with qualifiers (e.g. results apply only to novices)

The EBSE Evidence Map contains over 40 secondary studies, categorized using the SWEBOK classification. We also provide a growing list of highlighted secondary studies, and our bibliography lists over 140 studies, including more than 90 secondary studies.

What Research Opportunities does this provide?

We encourage researchers to undertake both mapping studies and systematic literature reviews as part of the task of mapping out our empirical domain knowledge. In addition, such studies, and especially mapping studies, whether undertaken by others or not, can provide the basis for identifying where new primary studies are needed, perhaps because no-one has studied a particular aspect, or because a replication would provide valuable confidence about the findings.

On this web site we provide information about the ideas and practices of evidence-based software engineering, identify experiences drawn from conducting such reviews and studies, and describe some of the outcomes.

One of the purposes of this site is to publish the experimental protocols (plans) as well as the results of conducting systematic literature reviews and other studies relevant to evidence-based practices in software engineering. We particularly encourage other researchers to register with the site and to provide material that can be shared with others, act as reviewers, and generally help form the community of evidence-based researchers in software engineering.

Page last updated on 11 June 2010.