PDF Version
Can we validly translate questionnaire items
Dale Griffee |
The purpose of this paper is to argue that items from a test instrument originally written in English, if translated into Japanese, cannot result in a valid test in Japanese simply by being translated. Researchers such as Shimizu (1995) as well as Widdows and Voller (1991) have had questionnaire items written originally in English translated into Japanese and then used data resulting from the translated items in order to make both statistical and qualitative inferences. None of the researchers gave reasons for the translation, although we can suppose that they wanted to ensure item comprehension by the Japanese participants in their studies. Their translated instruments were aimed at making inferences about students' needs or attitudes. Widdows and Voller wanted to know if their Japanese students' needs were being met by traditional teaching methods. Shimizu wanted to know if her students had different attitudes toward Japanese teachers of English as opposed to foreign teachers of English. Such research raises the issue of what it means for an instrument to have been validated. Put simply, to validate an instrument such as a test or a questionnaire, three things have to be done (for a more complete description see Griffee, 1997a). First, the construct has to be defined, which is to say "before developing a test of any construct, one should clearly and explicitly express what one wants to test" (Most and Zeidner, 1995, p. 482). This is sometimes known as defining your terms, but it probably should be known as defining your construct. Second, you have to give some evidence that the instrument has
consistency or reliability (Griffee, 1996). Third, the test maker has to provide some evidence that the test is measuring what the test maker claims it to be measuring (Brown, 1988, p. 101).
Three questions are raised in this paper. The first has to do with the validation of the original test instrument items. The second has to do with the possibility of the equivalence of the two instruments. The third question has to do with the validation of the L2 document. I will refer to the original English instrument as the E-doc and the resulting Japanese translation instrument as the J-doc. Terms such as "test", "instrument", and "questionnaire" are also used interchangeably in this article.
Three questions that every researcher working between languages should ask are:
[ p. 11 ]
|
1. Was the E-doc validated? 2. Is it possible for test items to be validly translated? 3. Was the J-doc validated? |
In many cross-linguistic research papers, constructs are not defined, reliability information is not provided, and no evidence is provided to show whether the instrument is measuring what it purports to. Though the papers by Shimizu (1995) and Widdows and Voller (1991) fail in each of
these respects, this is not unusual in our field in Japan. For example, as far as I know, no article with questionnaire data in The Language Teacher has ever
had adequate cross-linguistic validation information. Moreover, most articles in The JALT Journal with questionnaire data also lack this feature. If the examples supplied the original language were not validated carefully, the translation upon which the resulting documents was based cannot be considered valid.
Suppose that the E-doc were validated. Then what? Widdows and Voller themselves suggest the difficulty, if not impossibility, of a valid translation. They state, "It is interesting to note that certain concepts quite fundamental to current EFL methodology proved impossible to render into straightforward Japanese" (1991, p. 128). They add that another difficulty arose from Japanese cultural understanding dealing with an item asking about learning styles. The item wanted to know if the student learned better when the teacher took an interest in them as a person. The problem was with the word "interest" because "it was impossible to eradicate entirely the connotation of sexual interest in the Japanese version" (Widdows and Voller, 1991, p. 128).
Sakamoto (1996) investigated Hyland's (1994) use of translated questionnaire items originally from Reid's (1987) learning style preferences questionnaire. Sakamoto's participants were similar to the students used by Shimizu and Widdows and Vollers. The students were two groups of Japanese women aged 20 to 22 years of age at Bunka Women's University in Tokyo. Hyland had Reid's items translated and Sakamoto used these translated items, but she retranslated four of the items she thought were misleading. Sakamoto administered both the English items and the translated items to her students allowing time between administrations to reduce the possibility of remembering answers. She then compared the answers on the two questionnaires to see if the students answered the Japanese version differently than the English version and found that "about half of the 65 participants answered the same questionnaire statements differently in English and Japanese" (Sakamoto, 1996, p. 83). Sakamoto concludes:
Was the J-doc validated?[ p. 12 ]
Clearly, then, there were differences between the questionnaire results in English and Japanese. The high discrepancy in this study warms us that the researcher should not simply consider translation as the answer to help the respondent understand the questionnaire better. (p. 87).
Now comes the real kick in the head. I want to argue that the translated J-doc
becomes an instrument in itself and has to be validated as a separate instrument. The fact that the E-doc has been validated does not carry
over to the J-doc. We have to have evidence that the J-doc is measuring what it purports to be measuring. The mere fact that the students
speak the Japanese language as their L1 and that the J-doc has been translated into the Japanese language is not enough to provide validation.
If you are a native English speaker, you can ask yourself two questions. The first question is, "Have you ever read an English sentence which
was translated from Japanese (or another language) and been left wondering what it was trying to say?" The second question is, "Have you ever read an English sentence written originally in English by a writer you presumed to be an English L1 writer and still been left wondering what it meant?" Most of us can answer "yes" to both questions. That this is the case is exactly why validation of information is required in the first place. Even if the J-doc had been written originally in Japanese by a Japanese L1 writer, it would require validation.
I would like to conclude with two points. First, I am not trying to find fault with
the two instruments cited. When Widdows and Voller published their article I did not have the foggiest idea what reliability (much less
validity) was about. Furthermore, I was quite happy to write and have published my own unvalidated questionnaire (Griffee, 1997b).
Nevertheless, we are now more informed as to what is involved in the issues of reliability and validity. Second, we now have a
Testing & Evaluation SIG and a newsletter. I think that one of the uses we can put our
newsletter to is to discuss issues such as the one I have raised. In that sense I invite feedback, comments, and discussion on this article either for or against which can be sent to the editor of this publication.
[ p. 13 ]
ReferencesAuthor IndexTitle IndexDate Index
Background
Links
Network
Join
[ p. 14 ]