A Closed Hydraulic System
Based on some questions that were asked during today's class session, I can tell that some of you are still working to "pin down" the relationships among the different components of hypothesis testing. In an effort to help out, I'd like to describe (as best I can) an unusual "model" of hypothesis testing. If you're able now to create a mental image of something you've never seen before, then continue reading. If, however, your brain is too tired to do this, stop now, do something else for a few hours, and then return here once you feel up to the task of creative imagination.
I once was told that the hypothesis testing procedure functions like a "closed hydraulic system." Such a system, I was told, is nothing more than a set of pipes that lead to 5 expandable "vessels" where the pipe parts terminate. The entire system is completely filled with water. Because the system is "closed," water cannot escape. Moreover, any water that's forced out of any one of the expandable vessels necessarily must cause water to go into one or more of the other 4 vessels.
One of the expandable vessels in this closed hydraulic system represents the degree to which the sample data deviate from the null hypothesis. A second vessel represents the p-value that pops out of the computer after the data from the study have been analyzed.
Think exclusively about these two vessels and assume, for the time being, that the other three vessels are "held constant."
If the 1st of these 2 vessels (the one representing the degree to which the sample data deviate from Ho) is enlarged, then the "p" vessel must decrease in size. Thus, there's a clear and strong inverse relationship between the sizes of these 2 vessels. Increase one...and the other gets smaller. Make the data more in line with Ho and p goes up; make the data less consistent with Ho and p goes down.
It's dangerous to forget about the other 3 vessels, however, for small p-values (such as .003) do not necessarily indicate that there's a big discrepancy between the sample data and Ho.
Suppose we force almost all of the water OUT of the vessel that represents the discrepancy between the sample data and Ho. In other words, let's assume that the sample data are almost in COMPLETE agreement with the null hypothesis. Does this mean that p will be big and that we'll fail-to-reject Ho? Not necessarily!
In the situation we're now considering, it's conceivable that the p-vessel will be very small because most of the water has moved into one of the other 3 vessels. And that 3rd vessel might be the one which represents sample size. Thus, in a study with a large n, p can be small even though the sample data are only slightly different from Ho.
Hopefully, this discussion of the "closed hydraulic system" can help you see why it's a BIG MISTAKE to think that a study's p-value measures directly how large of an "effect" has been found. It doesn't. That's because there are other things that influence p (such as n), just as there are other vessels in the full hydraulic model.
In case you're wondering about the 4th and 5th vessels in the "closed hydraulic model," one represented the degree of within-group variability in the sample data. The other represents the one-sided vs. two-sided choice the researcher makes when setting up his/her alternative hypothesis.
I hope this "little" message helps. Specifically, I hope it encourages you to LOOK BEYOND THE p-LEVEL that's associated with a researcher's findings and examine carefully the actual size of the r (if the study deals with correlation) or at the raw differences between means (if the study involves a t-test or an F-test) or at the raw difference between percentages (if the study's data have been analyzed with a chi-square test). A p-level is important to consider, but it most assuredly DOES NOT provide, by itself, full insight into what's "going on" in a study.
Copyright © 2012
Schuyler W. Huck