In hypothesis testing, the size of a test is the (maximum) probability of committing a Type I error, that is, of incorrectly rejecting the null hypothesis when it is true.
Suppose that we are conducting a test about a parameter
that can take any value in a parameter space
.
The null hypothesis
is that
belongs to a given set
:
Denote by
the power function of the test.
For any
,
the function
gives us the probability of rejecting the null hypothesis when the true
parameter is
:![[eq5]](/images/size-of-a-test__11.png){kind=small}
Note that
depends on:
the true parameter
;
the null hypothesis
that we are testing;
the statistical procedure (test statistic and critical region) used to decide whether to reject the null.
The size of the test can be defined as follows.
Definition
The size of the test, denoted by
,
is
Thus, we consider all the cases in which the null is true
().
For each case, we compute the probability of (incorrect) rejection.
The size is equal to the largest value we find (worst-case scenario).
When the null hypothesis is simple, that is, the set
contains only one parameter (denote it by
),
the above definition
becomes
Thus, we have two cases:
if the null hypothesis does not specify an exact value for the parameter, but a whole set of parameters, then we need to take the maximum of the power function over that set in order to compute the size;
otherwise, if the null hypothesis specifies only one parameter, then it suffices to compute the value of the power function corresponding to that parameter.
Many authors use the term level of significance as a synonym for size.
For many others, the level of significance is an upper bound to the size, that
is, a constant
that
satisfies
For example, suppose that we are testing the null hypothesis that the mean
of a normal
distribution is equal to
.
The variance of the distribution, denoted by
,
is supposed to be known.
We observe a sample of
independent draws
from the distribution and we compute the
z-statistic![[eq12]](/images/size-of-a-test__28.png)
where
is the sample mean:
![[eq13]](/images/size-of-a-test__30.png)
We select a critical value
and reject the null hypothesis if
It can be proved (see
Hypothesis
testing about the mean) that the power function of the test
is![[eq15]](/images/size-of-a-test__33.png)
where
is the cumulative distribution
function of a standard normal random variable.
The size of the test
is![[eq16]](/images/size-of-a-test__35.png){kind=small}
By the
symmetry of
the standard normal distribution around
0,
we have
that![[eq17]](/images/size-of-a-test__37.png){kind=small}
As a consequence, we can write the size of the test
as
In order to better understand this result, consider that under the null
hypothesis the z-statistic has a standard normal distribution, that is, a
normal distribution with mean equal to
and variance equal to
.
If you set, for example,
,
then you will reject the null in two cases:
if the value of the z-statistic is less than
,
that is, if
;
if the value of the z-statistic is more than
,
that is, if
.
But we
know that if
has a standard normal distribution,
then![[eq19]](/images/size-of-a-test__47.png)
Thus the size of the test is 5%:
![[eq20]](/images/size-of-a-test__48.png){kind=small}
The following plot shows the probability density function of the z-statistic.
The black vertical segments indicate the two critical values.
When the value of the z-statistic falls in one of the two tails of the distribution (which are separated from the center of the distribution by the two critical values), the null hypothesis is incorrectly rejected.
The area under the probability density function in the two tails, colored with turquoise, is the probability of rejection, that is, the size of the test.
The area under the probability density function in the center of the distribution, colored with lavender, is the probability of acceptance.
In the previous example the size of the test was 5%. What if we want to decrease the size to 1%? How can this be achieved?
In general, the size of a test can be modified by changing the critical value(s) of the test, that is, by reducing or increasing the size (and hence the probability) of the critical region (remember that the critical region is the set of values of the test statistic that lead to rejection of the null hypothesis).
In the previous example, we can decrease the size of the test by increasing
the critical value
.
In particular, note that
implies
So, if our desired size is
,
then we need to search for what value the cumulative distribution function of
the normal distribution is equal to
.
By using normal distribution tables or a computer, we find that the desired
value is
.
As a consequence, the new critical value is
More details about the concept of size of a test can be found in the lecture entitled Hypothesis testing.
Previous entry: Score vector
Next entry: Standard deviation
Please cite as:
Taboga, Marco (2021). "Size of a test", Lectures on probability theory and mathematical statistics. Kindle Direct Publishing. Online appendix. https://www.statlect.com/glossary/size-of-a-test.
Most of the learning materials found on this website are now available in a traditional textbook format.
