Shiken: JALT Testing & Evaluation SIG Newsletter
Vol. 13 No. 2. May 2009. (p. 24 - 34) [ISSN 1881-5537]
PDF Version

Assessment Literacy Self-Study Quiz #7
by Tim Newfields

This ongoing column features questions about testing, statistics, and assessment in a quiz format to promote greater assessment literacy. Suggested answers to the problems below are online at http://jalt.org/test/SSA7.htm.

Part I: Open Questions

1. Please examine the following test item and then answer these questions:(1) What skills is this task likely tapping into? (2) In what ways (if any) might this item be inappropriate in a general university undergraduate EFL entrance exam?
   
   This item was taken from South Korea’s 2008 College Scholastic Ability Test [2008 Daehak suhakneungnyeok siheom] – one of the three types of exams that high school applicants seeking university admission must take.
   
   The task is to determine the correct sequence of three passages (A-C) that follow a 41-word paragraph opening.
   
   Source: Korea Institute of Curriculum and Evaluation. (2008). 2008 Haknyeondo Daehaksuhakneungryeoksihum Munjeji Oegukeo (Yeongeo) Yeongyeok – Holsuhyeong Oegukeo (Yeongeo) Yeongyeok 7. [2008 College Scholastic Ability Test Foreign Language (English) Field 7]. Retrieved April 7, 2009 from http://www.kice.re.kr/ko/board/view.do?article_id=64464&menu_id=10089&board_id=10153
   



[ p. 24 ]

2. Explain the difference between these concepts: (1) local test item independence and (2) test item unidimensionality.
   
   
3. Look at the box-percentile plots below, then briefly describe how the two groups differ. Also, what advantages do these projections have over common boxplots? Finally, how can such box-percentile plots such be drawn?
   
   
   Source: Harrell, F. & Banfield, J. (2005, March 8). R Graph Gallery Database: Box-Percentile plot. Retrieved April 5, 2009 from http://addictedtor.free.fr/graphiques/RGraphGallery.php?graph=77
   
   
4. In 2005 Hirsch proposed a simple index as one way of quantifying faculty research output and granting promotions. This index, known as the h-index, combines two metrics: (1) the number of papers one has written [k] and (2) the number of times those papers have written have been cited [d]. The precise formula is:
   
   Hence an author with a h-index of 12 has at least 12 publications, all of which have each been cited at least 12 times. That author might have a few publications with more citations, and many with less – but 12 is the largest number of articles-with-equal-number-of-citations for this person.
   
   What are some advantages and disadvantages of Hirsch’s ranking proposal?
   
   Source: Hirsch, J. E. (2005, September 1). An index to quantify an individual's scientific research output. Physics and Society, (DOI: 10.1073/pnas.0507655102) Retrieved April 3, 2009 from http://arxiv.org/abs/physics/0508025
   
   [ SUGGESTED ANSWERS ]

   [ p. 25 ]

   Part II: Multiple Choice Questions
   

   1. Which of the following statements is true about Poisson distributions?
      
      (A) They approximate binomial (discrete probability) distributions if the sample size is at least 200 and significance level is 0.01 or less. (B) As the number of events they describe decrease, they resemble normal bell-curve distributions more. (C) As the number of occurrences they describe increase, the sum of probabilities for each distribution approaches 0. (D) They are useful in predicting the frequency of periodic events.
      
   2. The graph below is an example of a (A) pareto chart (B) frequency polygon (C) cumulative distribution function (a.k.a ogive) (D) Q-Q plot (a.k.a quartile-quartile plot or QQ plot)
      
      
      
      
   3. Which of the following is least likely to improve the reliability of an examination?
      
      (A) lengthening the number of test items. (B) standardizing all conditions under which the test is administered, including instructions. (C) deleting any items from the test that did not correlate with other items. (D) creating multiple forms of the test.
      

      [ p. 26 ]

   4. One difference between the KR-20 and KR-21 is:
      
      (A) Only the KR-21 is robust if the unifactor trait is violated, provided the test is longer than 18 items. (B) Reliability estimates obtained from the KR-21 are generally lower than those obtained from the KR-20. (C) The former requires just one administration to measure a test’s internal consistency, but the later requires several administrations. (D) The former is used for dichotomous data, but the later is used for non-dichotomous data (i.e. partial credit answers).
      
      
   5. What is an ideal item facility index (a.k.a. “p-value” or “item difficulty index”) for a 4-option, multiple-choice norm-referenced test item? What about a true-false norm-referenced test item? What if both these items were designed for a criterion-reference test?
      
      NOTE: At least two answers for each sub-question are possible.
      (A) Probably between .4 and .6. (B) Probably around .63. (C) Probably around .75. (D) Probably above .8. (E) Actually, more information is needed before deciding that.
      
   [ SUGGESTED ANSWERS ]

   ---

   NEWSLETTER: Topic IndexAuthor IndexTitle IndexDate Index
   TEVAL SIG: Main Page Background Links Network Join

   ---

   
   HTML: http://jalt.org/test/SSQ7.htm   /   PDF: http://jalt.org/test/PDF/SSQ7.pdf
   

   ASSESSMENT LITERACY ONLINE SELF-STUDY SERIES:
   
   Quiz #1: Qs   / Qs   / As   *    2: Qs   / As   *    3: Qs   / As   *    4: Qs   / As   *    5: Qs   / As   *    6: Qs   / As   *    7: Qs   / As   *    8: Qs   / As   *    9: Qs   / As   *    10: Qs   / As   *    11: A HREF="SSQ11.htm">Qs   / As  

   ---

   [ p. 27 ]