A procedure (based on the Neyman–Pearson lemma) for deciding between two hypotheses on the basis of the value of a statistic called the test statistic, which is a function of the observations in a random sample. In a test concerning the value of an unknown parameter, the null hypothesis specifies a particular value for the parameter, whereas the alternative hypothesis specifies either an alternative value or, more usually, a range of alternative values. The null hypothesis is often denoted by H₀ or NH, and the alternative hypothesis by H₁ or AH.

A typical null hypothesis might state that the population mean μ = 20. The alternative hypothesis μ<20 is described as one-sided and the test procedure is described as one-tailed. By contrast, the alternative hypothesis μ≠20 is two-sided and the test is two-tailed.

If necessary, assumptions are made about the distribution type of the population, so that the probability distribution of the statistic can be determined assuming H₀. The probability of obtaining a value as extreme as the observed value or a more extreme value (taking account of the alternative hypothesis), is called the p-value. If the actual value of the statistic is too far from its expected value the test is deemed to be significant and the decision is to reject H₀ in favour of H₁. If the actual value of the statistic is close to its expected value the test is deemed to be not significant and the decision is not to reject H₀. The set of values of the statistic that lead to rejection of H₀ is called the critical region or rejection region, and the set of values that do not lead to rejection of H₀ is called the acceptance region.

There are two cases when the test leads to a correct result. These occur when H₀ is true and the test leads to its acceptance and when H₁ is true and the test leads to rejection of H₀. On the other hand there are two cases when the test leads to an incorrect result. These occur when H₀ is true but the test leads to its rejection (a Type I error, or error of the first kind) and when H₁ is true but the test leads to acceptance of H₀ (a Type II error, or error of the second kind).

The size of the critical region is determined by the desired significance level of the test, often denoted by α (alpha), which is the probability of making a Type I error. The smaller the significance level, the smaller the critical region. The significance level is usually expressed as a percentage. The word ‘significance’ seems to have been introduced into Statistics by Edgeworth in 1885. The ideas concerning the two types of error were introduced by Neyman and Egon Pearson in 1928.

As an example, suppose X has a normal distribution N(μ, 9) and it is desired to test H₀: μ = 20 against H₁: μ>20, using a sample of size 25. An appropriate statistic is the sample mean X̄, which has distribution N(20, 9/25) under H₀, or its standardized value which has distribution N(0, 1). If the desired significance level is 5%, the critical region, from tables of critical values for the standard normal distribution (see appendix iv) is Z>1.645. In terms of X̄, the critical region is X̄>20 + 1.645 or, equivalently, X̄>20.99.

The probability of making a Type II error is often denoted by β (beta). The power of the test, which is the probability of accepting the alternative hypothesis when it is in fact true, is 1−β and its value depends on the value of the parameter under test. A plot of the relation between power and the parameter value is called the power curve. If there is a choice of test, with a predetermined significance level, it is usual to choose the test (if one exists) that maximizes the power.

However, the variance may not be known. With the same assumption of a normal distribution, as before, a t-test is appropriate. Suppose (with the same hypotheses as previously) that the sample of 25 observations has sample mean and variance (using the (n−1) divisor) given, respectively, by x̄ = 21.4 and s² = 12.25. The test statistic t is given by The upper 5% point (see percentage point) of a t-distribution with 24( = 25−1) degrees of freedom is 1.711 (see appendix vi). Since 2.0>1.711, the null hypothesis would be rejected in favour of the alternative hypothesis.

Frequently, hypothesis tests involve the comparison of two populations. The case where unrelated random samples are taken from the populations gives rise to two-sample tests. A two-sample test of the equality of two population means in which the populations have the same (unknown) variance requires the use of a pooled estimate of the common variance (see pooled estimate of common mean). The case where the population variances are unknown and cannot be assumed to be equal is the Behrens–Fisher problem.

If the two random samples have the same size and matched pairs of individuals are obtained, one from each population, then, for a test of equality of population means, a paired-sample test is appropriate. Here the differences within each pair of values constitute the observations. When the data have a normal distribution (or the samples are large) a t-test is performed, the null hypothesis being that the mean difference is 0, and the sample variance being the variance of the differences.

A hypothesis that specifies several simultaneous conditions is described as a composite hypothesis. For example, with samples from k populations, the null hypothesis might specify that all k population means are equal.