Survey significance test calculator

How this tool works

Your four counts form a 2 × 2 table. For each cell the expected count under “no difference” is its row total × its column total ÷ the grand total, and the statistic is the sum of (observed − expected)² ÷ expected across all four cells, with one degree of freedom. For two proportions this is the same test as a two-sided z test on the difference; no continuity correction is applied.

For example, 120 of 400 in group A and 156 of 400 in group B gives 30% and 39%, a difference of 9 percentage points. The expected counts are 138 and 262 in each row. The chi-square statistic is about 7.2 and the p-value about 0.007.

A p-value is not the size of the effect and not the probability that you are right. It is how often chance alone produces a gap at least this large when there is nothing there. Read the percentage-point difference first and decide before you look whether that difference would change what you do. A result that clears 0.05 on 40 people can still be too imprecise to act on.

Choose comparisons before examining the results. Testing many questions increases the risk of false positives; report how many tests you ran and consider a multiple-testing adjustment. The tool warns when expected cell counts fall below five because the approximation may be unreliable.

The test assumes independent observations. A difference between self-selected groups does not establish a causal effect: people who choose a feature may differ in other ways. Random assignment can support a causal comparison when the study is designed appropriately. See how to run that inside a survey.