Why We Misjudge Our Own Effectiveness at Finding Software Bugs

This study investigates whether testers’ beliefs about which defect-detection technique works best for them actually predict real performance. In a controlled, crossover experiment within a Software Verification and Validation course, participants applied equivalence partitioning, branch testing, and code reading by stepwise abstraction across multiple C programs seeded with detectable faults. Researchers measured actual effectiveness, collected perceived effectiveness via a questionnaire, and quantified the “mismatch cost” and expected project loss when perceptions diverged from results, using agreement and statistical tests (e.g., kappa, Krippendorff’s alpha, chi-square, and ANOVA/Kruskal–Wallis).