The Bonferroni adjustment |
James Dean Brown University of Hawai'i at Manoa |
[ p. 23 ]
The problem is that Table 1 shows that six out of 100 comparisons are significant at .05 even though these analyses are based on data that are 100% random. I produced the data for each pair of groups to be a set of random normal standardized T scores. Yet, six of the t-tests turned out to be significant at .05. This is approximately what we would expect from random numbers. If I were to repeat the procedures with new random normal data, there might be four or three or seven or five significant t-tests, but on average over many such pseudo-studies based on random numbers, I would expect about 5% to be significant at p < .05. These are called spuriously significant results (i.e., significant differences that occur by chance alone). Such findings are completely reasonable because p < .05 indicates that there is a five percent probability of such findings by chance alone and indeed we found that approximately five percent (actually 6% in this particular instance) were indeed significant when based on random normal data.Test | Group 1 | Group 2 | Abs. Diff. | Abs. T | Test | Group 1 | Group 2 | Abs. Diff. | Abs. T | |||
T001 | 50.39 | 48.43 | 1.96 | 1.29 | T051 | 49.95 | 50.78 | 0.83 | 0.62 | |||
T002 | 51.68 | 49.62 | 2.06 | 1.40 | T052 | 50.53 | 49.60 | 0.93 | 0.64 | |||
T003 | 52.29 | 50.78 | 1.50 | 1.15 | T053 | 49.69 | 49.64 | 0.05 | 0.04 | |||
T004 | 50.25 | 48.56 | 1.69 | 1.16 | T054 | 50.45 | 51.63 | 1.18 | 0.85 | |||
T005 | 48.77 | 48.82 | 0.05 | 0.04 | T055 | 49.53 | 49.48 | 0.06 | 0.04 | |||
T006 | 51.72 | 50.75 | 0.96 | 0.74 | T056 | 51.35 | 50.00 | 1.35 | 0.96 | |||
T007 | 48.41 | 51.39 | 2.98 | 2.40 | * | T057 | 49.53 | 50.32 | 0.80 | 0.56 | ||
T008 | 50.65 | 49.90 | 0.75 | 0.52 | T058 | 48.13 | 50.04 | 1.92 | 1.33 | |||
T009 | 50.71 | 49.99 | 0.72 | 0.54 | T059 | 49.99 | 50.03 | 0.05 | 0.03 | |||
T010 | 50.60 | 49.31 | 1.29 | 1.01 | T060 | 51.89 | 50.11 | 1.78 | 1.28 | |||
T011 | 50.34 | 49.31 | 0.86 | 0.64 | T061 | 50.80 | 47.18 | 3.62 | 2.91 | * | ||
T012 | 50.78 | 48.98 | 1.79 | 1.20 | T062 | 51.61 | 49.99 | 1.61 | 1.10 | |||
T013 | 49.54 | 49.87 | 0.33 | 0.24 | T063 | 49.82 | 50.82 | 1.00 | 0.71 | |||
T014 | 50.37 | 51.07 | 0.70 | 0.52 | T064 | 50.94 | 49.90 | 1.04 | 0.73 | |||
T015 | 50.52 | 48.80 | 1.72 | 1.30 | T065 | 49.68 | 52.19 | 2.52 | 1.72 | |||
T016 | 52.13 | 49.68 | 2.46 | 1.82 | T066 | 50.75 | 51.75 | 1.00 | 0.66 | |||
T017 | 49.63 | 49.27 | 0.36 | 0.28 | T067 | 49.08 | 49.32 | 0.24 | 0.14 | |||
T018 | 48.95 | 51.45 | 2.50 | 1.62 | T068 | 49.19 | 50.76 | 1.56 | 1.09 | |||
T019 | 51.48 | 48.89 | 2.59 | 1.78 | T069 | 51.01 | 50.61 | 0.39 | 0.27 | |||
T020 | 50.80 | 48.82 | 1.97 | 1.34 | T070 | 49.99 | 50.67 | 0.68 | 0.48 | |||
T021 | 48.93 | 48.5 | 0.43 | 0.43 | T071 | 51.09 | 49.69 | 1.40 | 1.02 | |||
T022 | 51.26 | 49.79 | 1.47 | 1.01 | T072 | 49.46 | 49.01 | 0.45 | 0.32 | |||
T023 | 50.55 | 49.83 | 0.72 | 0.48 | T073 | 51.1 | 50.38 | 0.78 | 0.56 | |||
T024 | 49.65 | 51.33 | 1.68 | 1.19 | T074 | 50.47 | 49.37 | 1.10 | 0.78 | |||
T025 | 51.60 | 48.80 | 2.79 | 2.07 | * | T075 | 50.17 | 51.33 | 1.16 | 0.82 | ||
T026 | 51.07 | 51.39 | 0.32 | 0.23 | T076 | 49.72 | 49.88 | 0.16 | 0.11 | |||
T027 | 48.58 | 50.43 | 1.85 | 1.31 | T077 | 50.57 | 50.27 | 0.30 | 0.21 | |||
T028 | 51.10 | 50.72 | 0.39 | 0.28 | T078 | 48.72 | 46.85 | 1.87 | 1.41 | |||
T029 | 48.63 | 51.40 | 2.77 | 1.95 | T079 | 50.62 | 49.90 | 0.72 | 0.49 | |||
T030 | 50.35 | 49.73 | 0.62 | 0.43 | T080 | 50.98 | 49.15 | 1.84 | 1.42 | |||
T031 | 48.93 | 49.13 | 0.20 | 0.14 | T081 | 49.45 | 48.93 | 0.52 | 0.39 | |||
T032 | 49.80 | 51.16 | 1.36 | 0.93 | T082 | 51.10 | 51.10 | 0.00 | 0.00 | |||
T033 | 51.11 | 51.16 | 0.05 | 0.04 | T083 | 49.52 | 50.44 | 0.92 | 0.64> | |||
T034 | 50.45 | 49.40 | 1.05 | 0.74 | T084 | 49.87 | 50.18 | 0.30 | 0.22 | |||
T035 | 48.95 | 50.06 | 1.11 | 0.82 | T085 | 51.89 | 50.87 | 1.01 | 0.71 | |||
T036 | 52.09 | 49.87 | 2.22 | 1.50 | T086 | 49.31 | 49.72 | 0.41 | 0.32 | |||
T037 | 49.55 | 51.48 | 1.93 | 1.50 | T087 | 52.24 | 50.28 | 1.96 | 1.27 | |||
T038 | 49.03 | 50.05 | 1.02 | 0.67 | T088 | 49.92 | 51.34 | 1.42 | 1.05 | |||
T039 | 50.59 | 48.40 | 2.19 | 1.53 | T089 | 49.66 | 49.44 | 0.22 | 0.15 | |||
T040 | 49.45 | 49.32 | 0.13 | 0.08 | T090 | 49.76 | 49.28 | 0.49 | 0.33 | |||
T041 | 49.06 | 49.51 | 0.44 | 0.31 | T091 | 50.69 | 51.51 | 0.83 | 0.57 | |||
T042 | 50.70 | 49.01 | 1.69 | 1.16 | T092 | 52.07 | 48.40 | 3.67 | 2.77 | * | ||
T043 | 50.53 | 49.46 | 1.07 | 0.77 | T093 | 48.42 | 50.04 | 1.62 | 1.17 | |||
T044 | 49.66 | 50.96 | 1.30 | 0.89 | T094 | 47.99 | 50.81 | 2.83 | 2.01 | * | ||
T045 | 46.84 | 50.12 | 3.28 | 2.24 | * | T095 | 51.33 | 49.80 | 1.52 | 1.04 | ||
T046 | 49.02 | 49.78 | 0.77 | 0.55 | T096 | 50.47 | 49.69 | 0.79 | 0.59 | |||
T047 | 49.88 | 50.04 | 0.16 | 0.11 | T097 | 49.21 | 50.07 | 0.86 | 0.60 | |||
T048 | 49.82 | 49.17 | 0.66 | 0.44 | T098 | 48.74 | 49.13 | 0.39 | 0.26 | |||
T049 | 49.32 | 49.02 | 0.30 | 0.20 | T099 | 50.86 | 50.37 | 0.49 | 0.37 | |||
T050 | 49.58 | 50.28 | 0.71 | 0.52 | T100 | 49.01 | 49.10 | 0.09 | 0.06 |
[ p. 24 ]
[ p. 25 ]
[ p. 22 ]
[ p. 26 ]
[ p. 27 ]
Where to Submit Questions: |
Please submit questions for this column to the following address: |
JD Brown Department of Second Language Studies University of Hawai'i at Manoa 1890 East-West Road Honolulu, HI 96822 USA |
[ p. 28 ]