Shiken: JALT Testing & Evaluation SIG Newsletter
Vol. 15 No. 2. October 2011. (p. 20 - 22) [ISSN 1881-5537]
PDF PDF Version


* OPINION PIECE:

*Why isn't note taking allowed on the TOEIC®?*

by Edward Sarich


    In recent years, pressure to improve student performance on the TOEIC® has elicited an increase in university courses designed toward raising scores (McCrostie, 2006, p.30). The Educational Testing Service (ETS), the agency that produces this test, would have us believe that apart from familiarity with test item types, courses designed at specifically reviewing test questions are no more effective than general courses designed to improve English proficiency (Wood, 2010, p.41). However, the plethora of courses, textbooks, and training available to ready potential test takers for the TOEIC suggests that many disagree with this assertion.
    Although the research is rather scant, it has been suggested that an effective approach to teaching a TOEIC course focuses on three areas. First, as the grammar principles covered on this standardized test are not overly difficult, more time spent on vocabulary building activities might help raise test scores (Nishigaki & Chujo, 2005, p. 40). Second, structured group activities are thought to offer several advantages. Giving students a chance to work through problems with one another not only allows them to draw upon their individual strengths, it also offers an interesting alternative to the standard practice of reviewing sample test questions, the mundane and repetitive nature of which can negatively affect student motivation. (Davies, 2005).
    However, it is the third area of focus with which some concern is warranted. Meta-cognitive strategies are those that can help students become more aware of the effective application of their knowledge. Many of my Japanese university EFL students have expressed frustration in attempting to answer 100 listening questions on a 90-minute test. As most of them can only hope to get less than half the questions correct, the experience can be very discouraging, especially when there are several consecutive questions for which they have no answer. Similarly, getting "stuck" on certain questions can cause them to miss the audio for the next question as well, forcing them to guess randomly. There are even some who find the task of sitting through several snippets of spoken discourse ranging over a variety or random topics so overwhelming that many students experience a cognitive overload, then shut down and fall asleep. Some research suggests developing meta-cognitive skills, such as time management, focused concentration, and familiarity with test design, not only decreases simple errors, it can also lessen test anxiety, which in turn has been shown to increase test scores (Pan, 2010, p. 83). These performance issues are more closely associated with test taking than they are with language proficiency. By reducing or even eliminating these extraneous variables, more accurate and consistent results can be achieved, offering higher test reliability (Bachman & Palmer, 1996, p. 20).
". . . note taking can greatly reduce frustration among students because rather than having them take on the onerous task of trying to retain everything in memory, they are now actively trying to listen specifically for details which they deem to be relevant."

    One meta-cognitive strategy that I have employed in class is simple note taking on listening questions. Although listening is traditionally taught as a passive activity, it is now believed that good listeners are very active (Vandergrift, 1999, p. 168). Note taking can help students develop active skills, such as inferencing or activating the listener's experiential knowledge base (Wang, 2011, p. 361). It has also been my experience that note taking can greatly reduce frustration among students because rather than having them take on the onerous task of trying to retain everything in memory, they are now actively trying to listen specifically for details which they deem to be relevant. This is not just an effective strategy for test taking. Active listening in this way actually mimics authentic language activity, as native speakers are often required to practice focused listening, catching necessary information and filtering out that which is redundant (Ur, 1984, p. 91).

[ p. 20 ]

    Surprisingly, however, note taking during the TOEIC is prohibited, even though other commonly used standardized language test in Japan allows the practice, including the STEP-Eiken, the Sentaa Shiken, and the IELTS. Even the TOEFL, an internationally recognized test of English proficiency that is produced by the makers of the TOEIC, allows note taking.
    ETS explained that there were two reasons that note taking is forbidden (ETS, personal communication, June 29, 2011). First, not allowing test takers to write on anything other than answers on their answer sheets is thought to make it easier to spot cheaters. While this may or may not be true, it seems unfair to require test takers to limit their test taking strategies for this reason. In this regard, McNamara proposed that, "Invalidly low scores should not occur because the measurement contains something irrelevant that interferes with the affected persons' demonstration of competence" (1990, p. 111).
"One way in which tests can produce better learning outcomes is by fostering learning behavior that mimics authentic language activity. Allowing note taking, therefore would be of great benefit to the TOEIC . . ."

    The second reason ETS offers for not allowing note taking on the TOEIC is that it was designed to assess English in a 'business environment', which they claim is not likely to employ note taking. The TOEFL, they countered, is made to measure English in academia, an environment in which note taking is commonplace. However, while ETS might be able to broadly say that the TOEIC is a measure of English in business, they have no way of actually estimating how English will be used in that environment, therefore they cannot say with certainty that developing note taking skills will not benefit test takers. On the contrary, one can easily imagine situations in business, such as meetings or telephone conversations, in which note taking skills would be of great benefit. One way in which tests can produce better learning outcomes is by fostering learning behavior that mimics authentic language activity. Allowing note taking, therefore would be of great benefit to the TOEIC, which has been criticized for not adequately reflecting the language needs that test takers will require in business (Hirai, 2008, p. 8).
    But beyond the weak justification behind disallowing note taking, there is also the issue of how such a decision can negatively affect the validity of the TOEIC. Recent testing theorists now discuss validity not only in terms of how accurately a test is a reflection of the constructs it is attempting to assess, but of the consequences that the test produces (Bachman and Palmer, 1996, p. 34). Standardized tests such as the TOEIC have been accused of testing constructs that are only indirectly related to language ability, offering an unfair advantage for some test takers over others. Working memory, the organization and processing of short-term memory, is one such construct. Requiring test takers to rapidly shift modalities, as seen in the listening section of standardized language tests, is strongly affected by the facility of their working memory. (Aryadoust, 2011, p. 2). By prohibiting note taking, therefore, the makers of the TOEIC are offering an unfair advantage to those with a better working memory (Chapman & Newfields, 2008, p. 37). While innate constructs such as intelligence or superior working memory might indeed offer an advantage, including them in the evaluation of language raises many ethical implications, because it shifts the focus of assessment from language proficiency to one of language potential. It is easy to imagine the Orwellian consequences that might arise from allowing such constructs to be included in high stakes assessment.
    To some, this may seem like a small issue. There are undoubtedly those who do not believe that note taking is even beneficial for test takers. But the ETS's decision not to allow note taking on the TOEIC is symptomatic of a much larger problem. In the absence of any organization in existence to hold them accountable, ETS is capable of making decisions in their own interest and then justifying them by stating that they are for the benefit of the test taker. Without any evidence to support their claims, how are we to verify that what they say is true?

[ p. 21 ]

    Although teachers have traditionally been the arbiters of their students' fates, this duty has increasingly fallen into the hands of organizations with little vested interest in those who take their tests. Even making small changes to these mass-produced standardized tests can significantly affect the lives of a generation of young students. Clearly, greater accountability must be expected of these agencies. If teachers do not do so, who will?

References

ALC (2006). Intabyuu Houhou > Stage 2. Retrieved March 3, 2003 from http://www.alc.co.jp/edusys/sst/interview.html

Alderson, C. (2000). Assessing Reading. Cambridge, UK: Cambridge University Press.

Buck, G. (2001). Assessing Listening. Cambridge, UK: Cambridge University Press.

Chapman, M. (2005). A case study of the need for change in the language testing policies of a Japanese corporation. JLTA Journal (8). 51-67.

Douglas, D. (1992). Test of English for International Communication. In Kramer, J., & Conoley, J. C. (Eds.). The Eleventh mental measurements yearbook. Lincoln, NE: Buros Institute of Mental Measurements.

Educational Testing Service. (2004). TOEIC: Report on test takers worldwide. Retrieved April 10, 2007 from http://ets.org/Media/Research/pdf/TOEICTT03.pdf

Educational Testing Service. (2007). What's new about the new TOEIC test? Retrieved April 8, 2007 from http://www.ets.org/portal/site/ets/menuitem.c988ba0e5dd572bada20bc47c3921509/?vgnextoid=40e02cfcb983b010VgnVCM10000022f95190RCRD&vgne

Finster, G. (2004). What English do we teach our students? In A. Pulverness (Ed.), IATEFL 2003 Brighton Conference Selections (pp. 9-10). Canterbury: IATEFL.

Hirai, M. (2002). Correlations between Active Skill and Passive Skill Test Scores. Shiken: JALT Testing & Evaluation SIG Newsletter. 6(3), 2-8. Retrieved April 9, 2007 from http://jalt.org/test/hir_1.htm

[ p. 36 ]

Aryadoust, V. (2011). Application of the fusion model to while-listening performance tests. SHIKEN: JALT Testing & Evaluation SIG Newsletter, 15 (2), 2-9. Retrieved from http://jalt.org/test/PDF/Aryadoust2.pfd

Bachman, L. F., & Palmer, A. (1996). Language Testing and Practice. Oxford University Press.

Chapman, M. & Newfields, T. (2008), The 'New' TOEIC. SHIKEN: JALT Testing & Evaluation SIG Newsletter, 12 (2), 32-37. Retrieved from http://jalt.org/test/PDF/newTOEIC.pfd

Davies, A. (2005) A four-point approach to teaching TOEIC test preparation courses affectively and communicatively. The Language Teacher, 29 (2), 9-13. Retrieved from http://jalt-publications.org/tlt/issues/2005-02_29.2

Hirai, M. (2008, October 8). Engineers must have English skills to succeed. Japan Times. p. 8.

Hughes, A. (2003). Testing for language teachers. Cambridge: Cambridge University Press.

McNamara, T. (1990). Language testing. Oxford University Press.

McCrostie, J. (2006) Why are universities abandoning English teaching for TOEIC training? The OnCUE Journal, 14 (2), 30-32. Retrieved from http://jaltcue-sig.org/node/54

Nishigaki, C. & Chujo, K. (2005) Using CALL to bridge the vocabulary gap. Essential Teacher, 2 (4), 40-43. Retrieved from http://www5d.biglobe.ne.jp/~chujo/data/Essential_Teacher_Using_CALL_to_Bridge.pdf

Pan Y.C. (2010). Enhancing students' communicative competency and test-taking skills through TOEIC preparatory materials. TESOL Journal, 81 (3), 81-91.

Ur, P. (1984). Teaching listening comprehension. Cambridge University Press.

Vandergrift, L. (1999). Facilitating second language comprehension: acquiring successful strategies. ELT Journal, 53 (3), 168-176.

Wang, H. (2011). An exploration on designing college English listening class. Journal of Language Teaching and Research, 2 (2), 359-363.

Wood, D. (2010). TOEIC materials and preparation questions: An interview with an ETS representative. The Language Teacher 34 (6), 31-45. Retrieved from http://jalt-publications.org/files/pdf-article/art6.pdf

NEWSLETTER: Topic IndexAuthor IndexTitle IndexDate Index
TEVAL SIG: Main Page Background Links Join
Copyright (c) 2008 by Mark Chapman and Tim Newfields. All rights reserved.
last Main Page next
HTML: http://jalt.org/test/sar_1.htm   /   PDF: http://jalt.org/test/PDF/Sarich1.pdf

[ p. 22 ]