Evidence-Based Software Engineering

EBSE for...

About Evidence-Based Software Engineering and the EBSE Website

What is the Evidence-Based Paradigm?

The essence of the evidence-based paradigm is that of systematically collecting and analysing all of the available empirical data about a given phenomenon in order to obtain a much wider and more complete perspective than would be obtained from an individual study, not least because each study takes place within a particular context and involves a specific set of participants. While these factors can bias the outcomes from an individual study, taking a wider view should make it possible to produce more reliable conclusions and to minimise the effects of local factors.

Systematic Literature Reviews

The core tool of the evidence-based paradigm is the Systematic Literature Review (SLR). A systematic literature review (often abbreviated to systematic review) is an established research method that has been formulated to help with answering research questions in those domains where individual empirical studies are not easily controlled or replicated. It is a form of secondary study, aimed at gathering together and analysing all of the experimental results available for a topic in an objective, unbiased and consistent manner. (Within this context, a primary study is a single experimental study, which may take a range of forms including surveys, case studies, randomised experiments, quasi-experiments...)

Systematic reviews originated in clinical medicine, where two decades ago it was recognised that the outcomes from individual clinical trials were not a safe or sufficient basis for decision making. Although clinical medicine can make use of randomised controlled trials (RCTs), where techniques such as double blinding (neither the researcher nor the recipient know whether a 'treatment' or a placebo are being administered) help with objectivity, there are still many sources of possible bias within an individual study. Subsequently their use has spread into other domains where such experimental rigour is either difficult or impossible to achieve, and where many different experimental forms may well be employed.

(Click here for more information about Systematic Literature Reviews.)

How are Evidence-Based Practices employed for Software Engineering?

In clinical medicine, the widespread use of Randomised Controlled Trials, or RCTs, and the role of the human as a recipient of the experimental treatment, makes it possible to aggregate the different outcomes using statistical techniques. However, in domains where the human is a participant in empirical studies, and where individual skill and knowledge levels may influence their outcomes of a study, as occurs for software engineering, it is still possible to use the basic ideas of Systematic Literature Reviews, although usually these need to employ less statistically-rigorous forms of analysis.

Many Systematic Literature Reviews that have been published since the idea of adapting this paradigm for Software Engineering was first mooted in 2004.

A variation upon the Systematic Literature Review that is useful in software engineering is the mapping study (sometimes termed a scoping review). The purpose of this is to identify the set of primary studies available on a given topic, and in particular, to identify 'gaps' where more primary studies are needed as well as 'clusters' where there may be enough primary studies to warrant a fuller analysis. Mapping Studies generally have much more general research questions than an SLR, but use most of the practices (searching, inclusion/exclusion etc.) of a full review.

What does this site provide?

On this web site we provide information about the ideas and practices of evidence-based software engineering, identify experiences drawn from conducting such reviews and studies, and describe some of the outcomes. Whether you are a student, software developer, researcher or policy-maker with responsibility for software practices, there is material that you should find useful.

One of the purposes of this site is to publish the experimental protocols (plans) as well as the results of conducting systematic literature reviews and other studies relevant to evidence-based practices in software engineering. We particularly encourage other researchers to register with the site and to provide material that can be shared with others, act as reviewers, and generally help form the community of evidence-based researchers in software engineering.

Page last updated on 15 March 2012.