Article appearing in Shiken 17.1 (May 2013) pp. 27-32.
Author: James Sick
International Christian University, Tokyo
Introductory Paragraphs:
Previous installments of this series have provided an overview of Rasch measurement theory, reviewed the differences among the various Rasch models, and discussed the assumptions and requirements that underlie Rasch measurement theory (RMT). In this installment, I will address a practical problem that can occur when using many-facet Rasch analysis (MFRA). MFRA is often used to adjust for differences in rater severity or other factors when measures are constructed from subjective judgments. Readers unfamiliar with MFRA and the differences among the Rasch family of models might wish to review Part 3 in this series.
Question:
My institution recently held a student speech contest with 9 teachers serving as volunteer judges. The 51 student participants were assigned to 3 rooms where a three-judge panel rated each speech for content, language, and presentation. When all speeches were completed, the scores were compiled and the three highest scoring students received a prize.
Now that the contest has finished, I am analyzing the results with MFRA with the aim of improving the judging process in future contests. When I run the analysis using Facets (Linacre, 2012a), it runs but returns the message "warning -- there may be 3 disjoint subsets." Could you explain what this means and what, if anything, I should do about it?