Combinatorial testing is a difficult problem. Having to test a small number of inputs can result in a combinatorial explosion of possible permutations. In Cucumber we see this problem in Scenario Outlines where we can have a large number of rows for the Examples table.
We want to reduce the combinations to a more manageable size while still providing effective fault detection.
Pairwise testing provides one method of achieving this. However providing effective fault detection is dependent on the suitability of Pairwise to the data/system, it is not guaranteed.
What is Pairwise testing?
Pairwise testing (also called All-pairs testing or 2-way testing) is a way of generating a test suite which covers all input combinations of two and is therefore much smaller than exhaustive suites.
To put this in perspective a system of 72 binary inputs with Pairwise testing would require 28 combinations. Exhaustive testing would require 37,778,931,862,957,161,709,568 combinations!
Sounds great, but how does the method preserve good defect detection?
Pairwise relies on a simple principle:
“most faults are caused by interactions of at most two factors” – The Combinatorial Design Approach to Automatic Test Generation
Pairwise focuses on the minimal set of inputs (1 or 2 interactions) that cover the most likely causes of faults.
So what’s the catch?
Pairwise does not guarantee coverage of all faults of at most two factors. It only provides coverage of faults reachable from the inputs and values you select when generating the Pairwise set. Also, the general principle that faults are caused by at most two factors does not guarantee that your data sets faults will follow that distribution. And hence Pairwise testing might miss a proportion of faults.