Shiken: JALT Testing & Evaluation SIG Newsletter
Vol. 13 No. 3. Nov 2009. (p. 26 - 28) [ISSN 1881-5537]
PDF Version

Part I: Open Questions

1. An article by Higginbotham (2009, pp. 15-18) in a recent issue of The Language Teacher attempted to highlight the value of extended task based instruction. Support for this instructional approach was based on a single questionnaire about a carol singing activity using that instructional framework. One month after the activity, 69 of of the 80 students who participated in that activity completed a Likert questionnaire and vocabulary test with merely five words from the Christmas carols they had supposedly memorized. Based on those results, the author suggests activities such as this lead to "deep processing of the language" (p. 15).
   
   What confounding issues are present in this study? How could the validity and reliability of studies such as this be enhanced? Also, how should the graph to the right probably have been revised for academic publication?
   
   Source: Higginbotham, G. (2009). Event-based learning: The benefits of positive pressure. The Language Teacher. 33 (1) 15-18.
   



[ p. 26 ]

2. At most (if not all) universities in Asia, multiple entrance pathways exist. One widely used pathway is the entrance by recommendation system In this system, a limited number of students from a select number high schools are allowed to recommend canditates for admission to a specific university. Those select schools are often, but not always, affiliated with each given university in some way. In many cases, admission decisions are based on a submitted essay and a very short screening interview. Applicants generally have a month or so to compose their essays and they usually solicit the help of parents and/or teachers when composing and revising their essays. Moreover, most of the oral interview questions are known in advance and many students are able to churn out memorized answers to the most common questions.
   
   What specific test validity issues are compromised through this process? What viable alternatives to the current entrance by recommendation (suisen nyuushi) system exist?
   
   
3. What are the pros and cons of balancing the correct answer choice sequences in the answer sheets of fixed-response exams? What alternative ways of organizing multiple-choice key answers currently exist?
   
   Source: Harrell, F. & Banfield, J. (2005, March 8). R Graph Gallery Database: Box-Percentile plot. Retrieved April 5, 2009 from http://addictedtor.free.fr/graphiques/RGraphGallery.php?graph=77
   


4. In an ETS-funded research report, the TOEIC Bridge (Reading Section) was found to have a .84 correlation with a government run test of mathematics ability among 198 high school students in Columbia as the graph below suggests. What would be an appropriate interpretation of these results?
   
   Source: Sinharay, S. et al. (2008). Establishing the Validity of TOEIC Bridge Test Scores for Students in Colombia, Chile, and Ecuador. ETS Research Report 08-58. Retrieved on November 12, 2009 from http://www.ets.org/Media/Research/pdf/RR-08-58.pdf
   
   [ SUGGESTED ANSWERS ]

Part II: Multiple Choice Questions

1. One source of confusion in the field of language testing is that the symbol p is used to denote two entirely different concepts. Which concepts below are commonly signified by p by different authors?
   
   (A) item difficulty (also symbolized as ID). (B) the probably of a chance occurrence (C) the population size (D) They are useful in predicting the frequency of periodic events.
   

[ p. 27 ]

2. Which of the following statements are true about the KR-21? (Note: More than one correct response is possible.)
   (A) It is a good estimate of the KR20 if the range of item difficulty is relatively narrow. (B) It requires only one test administration. (C) Its reliability estimate is generally lower than the KR-20's. (D) It is robust if the unifactor trait is violated, provided the test is longer than 18 items.
   
   
3. Which of the following statements are not true about Wald tests? (Note: More than one correct response is possible.)
   
   (A) They can be used for dichotomous and continuous variables. (B) They provide a maximum likelihood estimate of a parameter(s) of interest. (C) They are generally preferred to likelihood ratio tests. (D) It does not require a large sample size in cases MANOVA models
   
4. The following grading scheme suggested by Davies (2002) is an example of (A) confidence marking (B) elimination testing (C) a liberal multiple-choice test grading scheme (D) an order-of-preference grading scheme
   
5. Which of the following statements are true about parameter drift? (Note: More than one correct response is possible.)
   
   NOTE: At least two answers for each sub-question are possible.
   (A) It may be caused by repeated exposure to a given test item. (B) It may be caused by a change in pre-test/post-test motivational levels. (C) It can be estimated by marginlikelihood model. (D) It can be estimated by a Bayes modal procedure .
   

---

NEWSLETTER: Topic IndexAuthor IndexTitle IndexDate Index
TEVAL SIG: Main Page Background Links Network Join

---

HTML: http://jalt.org/test/SSQ8.htm   /   PDF: http://jalt.org/test/PDF/SSQ8.pdf

ASSESSMENT LITERACY ONLINE SELF-STUDY SERIES:

Quiz #1: Qs   / As    *    2: Qs   / As    *    3: Qs   / As     *    4: Qs   / As    *    5: Qs   / As   *    6: Qs   / As    *    7: Qs   / As    *    8: Qs   / As    *    9: Qs   / As   *    10: Qs   / As   *    11: A HREF="SSQ11.htm">Qs   / As  

---

[ p. 28 ]