>>Return to Tell Me About Statistics!
In clinical studies, instead of performing the test only once, multiple comparisons may need to be made, such as “comparing drugs X, Y, and Z.”
In this article, we will discuss the multiple comparison method.
What is multiple comparison
When comparing “drug X, drug Y, and drug Z,” if two groups are evaluated at a time, like a t-test, the test must be performed three times: “drug X and drug Y,” “drug Y and drug Z,” and “drug X and drug Z.”
As explained in Vol. 23, the probability of causing an αerror is 5%.
What is the probability of getting an α error one out of three times?
The probability that an α error does not occur thrice is the cube of the probability that an α error does not occur once. The cube of “1 – 0.05 = 0.95” gives 0.857. In other words, the probability of having an α error even once out of three is 1-0.857 = 0.143 = 14.3%.
Compared with 5%, this percentage is significant. In other words, the more the test is repeated, the more likely it is that an αerror will occur.
The probabilities of occurrence of error are 22.6% for five tests, 40.1% for ten times, and 53.7% 15 times, which cannot be ignored. If we repeat comparisons like “drug X, drug Y, drug Z, and …”, even if there is no difference at all, there is a considerable possibility that some combination of drugs will show a significant difference.
Therefore, “the problem of the possibility of α error increasing as a result of repeated comparisons” is called multiple comparison.
Multiple comparison method
The multiple-comparison method is “a test method that can address the problem of multiple comparisons.” There are various methods for multiple comparisons; however, we have introduced the simplest method.
Repeating the test at the 5% significance level increases the likelihood of an α error. Thus, the significance level can be lowered to maintain a low probability. “Bonferroni” proffers a solution to lower significance level.
Bonferroni is “a method to test the mutual difference in population means between two groups for three or more populations.”
Bonferroni significance level
How to calculate the Bonferroni significance level.
[Bonferroni significance level].
= 5% ÷number of group combinations to be compared
*Number of groups = K
*Number of combinations = K ( K – 1) ÷ 2
For example, calculating the significance level when the number of groups is three,
・Number of groups = K = 3
・Number of combinations = 3 x 2 ÷ 2 = 3
Significance level = 5% ÷ 3 = 1.67%
The advantage of the Bonferroni method is that it can be used as is, for just correcting for the significance level. However, compared to other multiple comparison methods, it has been observed to be a slightly stricter test, making it difficult to obtain a significant difference.
>>Return to Tell Me About Statistics!
Comments are closed