Metamorphic testing

Metamorphic Testing (MT) [1] [2] [3] [4] [5] [6] is a software testing technique that attempts to alleviate the test oracle problem. A test oracle is the mechanism by which a tester can determine whether a program has failed. A test oracle problem occurs when it is difficult to determine the expected outcomes of selected test cases or to determine whether the actual outputs agree with the expected outcomes.

MT recommends that, given one or more test cases (called "source test cases" or "original test cases") and their expected outcomes (if any), one or more follow-up test cases can be constructed to verify the necessary properties (called "metamorphic relations" or "MRs") of the system or function to be implemented. Consider, for example, a program that implements sin x correct to 100 significant figures. A useful metamorphic relation for sine functions is "sin (π − x) = sin x", Thus, even though the expected value of sin x1 for the source test case x1 = 1.234 correct to the required accuracy is not known, a follow-up test case x2 = π − 1.234 can be constructed. We can verify whether the actual outputs produced by the program under test from the source test case and the follow-up test case are consistent with the MR in question. Any inconsistency indicates a failure of the implementation.

In general, an MR does not need to be an equation but can be a relation. Metamorphic testing can be applied to any problem in which a necessary property involving multiple executions of the target function can be formulated. Some examples of successful applications include partial differential equations,[7] ubiquitous computing,[8] services computing,[9] imaging software,[10] bioinformatics,[11] software product lines,[12] and web search engines.[13]

References

  1. ↑ First introduced in T.Y. Chen, S.C. Cheung, and S.M. Yiu, "Metamorphic testing: a new approach for generating next test cases", Technical Report HKUST-CS98-01, Department of Computer Science, Hong Kong University of Science and Technology, Hong Kong (1998).
  2. ↑ First regular conference paper: T.Y. Chen, T.H. Tse, and Z. Zhou, "Fault-based testing in the absence of an oracle", in Proceedings of the 25th Annual International Computer Software and Applications Conference (COMPSAC '01), IEEE Computer Society, pp. 172–178 (2001).
  3. ↑ First journal paper: T.Y. Chen, T.H. Tse, and Z.Q. Zhou, "Fault-based testing without the need of oracles", Information and Software Technology, vol. 45, no. 1, pp. 1–9 (2003).
  4. ↑ First IEEE TSE paper: T.Y. Chen, T.H. Tse, and Z.Q. Zhou, "Semi-proving: an integrated method for program proving, testing, and debugging", IEEE Transactions on Software Engineering 37 (1): 109-125 (2011).
  5. ↑ Recent IEEE TSE paper: Z.Q. Zhou, S. Xiang, and T.Y. Chen, "Metamorphic testing for software quality assessment: A study of search engines", IEEE Transactions on Software Engineering, 42 (3): 264-284 (2016).
  6. ↑ Survey paper: S. Segura, G. Fraser, Ana B. Sánchez and A. Ruiz-Cortés, "A Survey on Metamorphic Testing", IEEE Transactions on Software Engineering, DOI: 10.1109/TSE.2016.2532875, 2016.
  7. ↑ T.Y. Chen, J. Feng, and T.H. Tse, "Metamorphic testing of programs on partial differential equations: a case study", in Proceedings of the 26th Annual International Computer Software and Applications Conference (COMPSAC '02), IEEE Computer Society, pp. 327−333 (2002).
  8. ↑ T.H. Tse, S.S. Yau, W.K. Chan, H. Lu, and T.Y. Chen, "Testing context-sensitive middleware-based software applications", in Proceedings of the 28th Annual International Computer Software and Applications Conference (COMPSAC '04), vol. 1, IEEE Computer Society, pp. 458−465 (2004).
  9. ↑ W.K. Chan, S.C. Cheung, and K.R.P.H. Leung, "A metamorphic testing approach for online testing of service-oriented software applications", International Journal of Web Services Research 4 (2): 60−80 (2007).
  10. ↑ R. Guderlei and J. Mayer, "Towards automatic testing of imaging software by means of random and metamorphic testing", International Journal of Software Engineering and Knowledge Engineering 17 (6): 757−781 (2007).
  11. ↑ T.Y. Chen, J.W.K. Ho, H. Liu, and X. Xie, "An innovative approach for testing bioinformatics programs using metamorphic testing", BMC Bioinformatics 10: article no. 24 (2009).
  12. ↑ S. Segura, R.M. Hierons, D. Benavides, and A. Ruiz-Cortes, "Automated test data generation on the analyses of feature models: a metamorphic testing approach", in Proceedings of the 3rd International Conference on Software Testing, Verification, and Validation (ICST '10), IEEE Computer Society, pp. 35−44 (2010).
  13. ↑ Z.Q. Zhou, S. Zhang, M. Hagenbuchner, T.H. Tse, F.-C. Kuo, and T.Y. Chen, "Automated functional testing of online search services", Software Testing, Verification and Reliability 22 (4): 221−243 (2012).
This article is issued from Wikipedia - version of the 6/5/2016. The text is available under the Creative Commons Attribution/Share Alike but additional terms may apply for the media files.