Validation of mutation testing in the safety critical industry through a pilot study

Mutation testing is the state-of-the-art technique to evaluate the fault-detection capabilities of a test suite, but its adoption has been limited. In this paper, we aim to investigate where mutation testing fits within the existing test strategies, whether the mutation analysis can be done in a timely manner, and which pain points remain to be tackled for industrial integration. For this, we performed open format interviews with two companies developing safety critical software. Our first case had no experience with mutation testing, allowing us to analyse and aid them with the setup, integration, and the mutation analysis. Our second case has 5 years of experience with mutation testing, providing us with a mature view on mutation testing in practice. Our study found that mutation testing can be combined with continuous integration and offload the work of the human reviewer by providing an initial code-quality review. Equivalent mutants appear to be a less prevalent obstacle, although flaky mutants are a concern that needs to be addressed. Overall, the industrial perception of mutation testing is evolving as more organisations recognise the potential benefits of the technique and work to address its limitations and challenges.

ISBN:979-83-503-3335-0

Publication year:2023

Keywords:P1 Proceeding

Accessibility:Open