Evidence-Based Software Engineering

EBSE for...

Secondary Studies


Improving Evidence about Software Technologies: A Look at Model-Based Testing

Author(s)Neto, A.D., Subramanyan, R., Viera, M., Travassos, G.H. and Shull, F.
AbstractModel-based testing (MBT) approaches help automatically generate test cases using models extracted from software artifacts, and hold the promise to greatly affect how we build software. A review of the literature shows that certain specialized domains are applying MBT, but it does not yet seem to be a mainstream approach. The authors therefore conducted a systematic review of the literature to investigate how much evidence is available on MBT's costs and benefits, especially regarding how these techniques compare to other common testing approaches. They use these results to derive suggestions regarding what types of studies might further increase the deployment of these techniques.
DetailsNeto et al. found 85 papers discussing Model-based testing but the majority were based on speculation (23 studies), examples (38 studies), proof of concept (13 studies). In terms of studies providing empirical evidence, there were 4 industrial experience reports and 7 experiments. The 7 experiments were relatively small scale and all except one, concentrated on the percentage of seeded defects found. The exceptional experiment compared subjects (undergraduates) using a model-based technique with an extended finite state machine method and found that the model-based technique took longer to use and the subjects found it harder to use, although it performed better. They conclude that some issues make MBT deployment difficult (the quality or not of the models, the frequent lack of support for non-functional constraints) but "such approaches have benefits that might warrant the necessary training and cost".