Statistical Significance

Statistical significance is one of the necessary conditions a A/B test must achieve prior to calling it a winner in frequentist hypothesis testing. Statistical significance, contrary to its intuitive meaning, denotes the probability of achieving the results of the test by chance. Significance of 95% means there is only 5% chance that test results could have been achieved by pure chance. It is used to ensure the test does not make a Type I Error. While reaching significance is important when you conduct a test, it is not sole condition that allows you to call the test. Test must run until it reaches significance and pre calculated sample size. Otherwise, you risk a type II error or false negative, invalidating your test result.