Post Hoc Analysis for an F-Test

Solution Preview

Post hoc comparisons
What are post hoc comparisons, and why do researchers make them? Why do researchers not use the Bonferroni procedure of procedures for post hoc comparisons? What is the advantage over the Bonferroni procedure of procedures such as the Tukey and Scheffe tests? Is Scheffe preferred over others if so, why and what are the advantages and disadvantages? How do you carry out the Scheffe procedure?
Answer
The ANOVA uses the F test to determine whether there exists a significant difference among treatment means or interactions. In this sense it is a preliminary test that informs us if we should continue the investigation of the data at hand. If the null hypothesis (no difference among treatments or interactions) is accepted, there is an implication that no relation exists between the factor levels and the response. There is not much we can learn, and we are finished with the analysis. When the F test rejects the null hypothesis, we usually want to undertake a thorough analysis of the nature of the factor-level effects.
When processes are compared and the null hypothesis of equality (or homogeneity) is rejected, all we know at that point is that there is no equality amongst them. But we do not know the form of the inequality. Post hoc comparisons are generally performed after obtaining a significant omnibus F. Then we look at all possible pairwise or all possible pairwise and otherwise comparisons. Here, we are focusing on the largest difference between levels of the IV, but you are still sifting through the data in hopes of finding something significant.
There are two varieties of post hoc techniques: 1) pairwise; and 2) pairwise and otherwise. In pairwise techniques, we are only comparing two means at a time. All comparisons involve only two means or totals at a time. In the pairwise and otherwise techniques, minimally one comparison involves more than two conditions (i.e. group 1 + 2 vs. 3). In an experiment with "a" levels of an IV, there are ...