Shiken: JALT Testing & Evaluation SIG Newsletter
Vol. 15 No. 2. October 2011 (p. 30 - 42) [ISSN 1881-5537]
PDF Version

Part I: Open Questions

1. At most universities in Japan, how do the AO entrance examinations tend to differ from regular entrance exams and the National Center Test?
   
   
2. What is a "false consensus effect" (FCE)? How might it skew some social science data and how should it be avoided?
   
   
3. What unstated assumptions are inherent in the TOEIC "Can Do" statement chart below? What issues need to be considered when interpreting this chart? How could this information be more ethically presented?

Source: Ingles Personalizado. (2011). TOEIC. Retrieved from http://ingles-personalizado.com/toeic

4. Illustrate how a study with a regression discontinuity design could be structured to ascertain the effects of a specific intervention on English language reading fluency among a group of EFL high school students.
   
   

[ p. 30 ]

5. What additional information should have been mentioned in the chart below to help readers to interpret the data more meaningfully? Also, how would you interpret this data distribution?

Source: Northwood, B. & Thompson, C. K. (2010). Why stop studying Japanese? A case in Australia. In A. M. Stoke (Ed.) JALT 2009 Conference Proceedings. (p. 821). Retrieved from http://jalt-publications.org/archive/proceedings/2009/E040.pdf

6. In factor analysis, how does a communality differ from a commonality?
   

Part II: Multiple Choice Questions

1. A tendency of respondents to concur with the opinions implicitly expressed in a given rating scale (and avoid offering counter-opinions) is a known as ________________.
   
   (A) contamination bias (B) acquiescence bias (C) social desirability bias (D) central tendency bias (E) nonresponse bias
   
NOTE: One of these choices is most relevant; two are somewhat relevant, and two are irrelevant.


2. IWhat is one difference between the standard error of measurement (SEM) and standard error of difference (SED)??
   
   (A) The former is a test of significance between two individual test scores. (B) The latter indicates how widely an individual's given test score is likely to differ from his/her true score. (C) The latter is the critical value of a t-test of the difference between two means. (D) To calculate the latter, the former must be known.
   
   
NOTE: Just one answer is considered fully correct.


3. What sort of test would be most appropriate to ascertain whether a large sample of male and female university students differed significantly in their attitudes towards study abroad according to a Likert scale with 4 or 5 options??
   

   [ p. 31 ]

   (A) An unpaired t-test (B) An ANCOVA (C) An ANOVA (D) A MANOVA (E) A chi-square test
   
   
NOTE: : Two answers are viable, but one answer will yield more information than the other.


4. Which of the following points is not problematic about the avowed ranking of these universities in the following chart?
   
   
   Source: 2011 University Web Ranking. (2011). Universities in Japan. Retrieved from http://www.4icu.org/jp/
   (A) The criteria for the ranking is not explained in the title of the chart or anywhere on the same page as the chart. (B) This ranking was made entirely on the basis of a secret web metric formula - presumably, universities with lots of web traffic according to some search engines will receive higher ratings than those with low web traffic. (C) The fact that the different faculties at the same university may have different rankings is ignored. (D) The point that the lesson quality might vary widely from teacher to teacher at the same school is also ignored. (E) The location of some of the schools appears to be incorrect. (F) There is no mention of the measurement error of this rubric or indication of how widely apart the various school rankings are. Moreover, the possibility that two schools may have essentially equivalent rankings according to a given criterion does not appear to be considered.
   

---

NEWSLETTER: Topic IndexAuthor IndexTitle IndexDate Index
TEVAL SIG: Main Page Background Links Join

---

HTML: http://jalt.org/test/SSQ11.htm   /   PDF: http://jalt.org/test/PDF/SSQ11.pdf

---

ASSESSMENT LITERACY ONLINE SELF-STUDY SERIES:

Quiz #1: Qs   / Qs   / As   *    2: Qs   / As   *    3: Qs   / As   *    4: Qs   / As   *    5: Qs   / As   *    6: Qs   / As   *    7: Qs   / As   *    8: Qs   / As   *    9: Qs   / As   *    10: Qs   / As   *    11: Qs   / As  

---

[ p. 32 ]