Syntactic and Semantic Differencing for Combinatorial Models of Test Designs

Rachel Tzoref-Brill, Shahar Maoz

Research output: Chapter in Book/Report/Conference proceedingConference contributionpeer-review

Abstract

Combinatorial test design (CTD) is an effective test design technique, considered to be a testing best practice. CTD provides automatic test plan generation, but it requires a manual definition of the test space in the form of a combinatorial model. As the system under test evolves, e.g., due to iterative development processes and bug fixing, so does the test space, and thus, in the context of CTD, evolution translates into frequent manual model definition updates. Manually reasoning about the differences between versions of real-world models following such updates is infeasible due to their complexity and size. Moreover, representing the differences is challenging. In this work, we propose a first syntactic and semantic differencing technique for combinatorial models of test designs. We define a concise and canonical representation for differences between two models, and suggest a scalable algorithm for automatically computing and presenting it. We use our differencing technique to analyze the evolution of 42 real-world industrial models, demonstrating its applicability and scalability. Further, a user study with 16 CTD practitioners shows that comprehension of differences between real-world combinatorial model versions is challenging and that our differencing tool significantly improves the performance of less experienced practitioners. The analysis and user study provide evidence for the potential usefulness of our differencing approach. Our work advances the state-of-The-Art in CTD with better capabilities for change comprehension and management.

Original languageEnglish
Title of host publicationProceedings - 2017 IEEE/ACM 39th International Conference on Software Engineering, ICSE 2017
PublisherInstitute of Electrical and Electronics Engineers Inc.
Pages621-631
Number of pages11
ISBN (Electronic)9781538638682
DOIs
StatePublished - 19 Jul 2017
Event39th IEEE/ACM International Conference on Software Engineering, ICSE 2017 - Buenos Aires, Argentina
Duration: 20 May 201728 May 2017

Publication series

NameProceedings - 2017 IEEE/ACM 39th International Conference on Software Engineering, ICSE 2017

Conference

Conference39th IEEE/ACM International Conference on Software Engineering, ICSE 2017
Country/TerritoryArgentina
CityBuenos Aires
Period20/05/1728/05/17

Fingerprint

Dive into the research topics of 'Syntactic and Semantic Differencing for Combinatorial Models of Test Designs'. Together they form a unique fingerprint.

Cite this