Grading student oral test performance: a closer look at the data



In the Shiken article accompanying this, the rational behind the development of a classroom specific analytical rating scale for small group oral tests was outlined. The resulting scales (included below) included three parameters of assessment: content, communication skills, and English skills.

GRADE: C LEVEL B LEVEL A LEVEL
CONTENT: demonstration of some comprehension of topic and ongoing discourse as well as minimal contribution to that topic and discourse demonstration of preparation on topic (i.e. did the student come prepared with something meaningful to contribute?) demonstration of some detail and depth in discussion of topic and a chosen topic of relative complexity for a false beginner
COMMUNICATION SKILLS: demonstration of interest and basic communication skills (i.e. expressions or indications of interest and eye contact) demonstration of active listening (i.e. did the student respond verbally or otherwise to partner utterances) and the ability to communicate most intendedÊ meanings via any of a variety of communication skills (verbal or nonverbal) demonstration of the succinct communication of intended meanings, the effective elicitation of meaning, and strategy in overcoming communication breakdowns
ENGLISH SKILLS: demonstration of ability to correctly formulate and interpret some simple expressions of meaning (i.e. asking and answer yes/no questions demonstration of adequate lexical and structural knowledge to correctly formulate and interpret many basic meanings demonstration of broader lexical and structural knowledge to allow for the confident, succinct, and accurate expression and interpretation of most basic meanings as well as some more complex meanings


The crucial question is the extent to which the scales form the real, explicit rationale behind teacher grading of a given elicited distribution of performances. This requires a closer look at the elicited data and rater decisions in response to that data. What follows is a selection of transcripts taken from oral tests and a step by step description of grading decisions. The students were tested in groups of three and were given approximately four minutes to converse on a topic randomly selected from a total of four. The student were given prior notice of the topics which were extensively covered in regular classes through the semester. The proceeding transcribed excerpt was taken from an oral test under the topic heading of "superstitions".

Transcript Key
(.)   =   short pauses of less than a second ...   =   longer pauses of over a second
[   =   simultaneous speech =   =   overlapped speech without a pause


1) S1 I don't believe (.) this kind of superstition. I believe like... uh... ghost

2) S3: Ghost?

[

3) S2: Ghost?

4) S1: Yeah.

5) S3: Really?

6) S1: Yes.

[

7) S2: I believe too.

8) S3: Really?

9) S1: Yeah. Yeah.

[

10) S2: Have you ever ...ever feel uh something ...(gesture behind himself) invisible?

11) S1: (laughs)

[

12) S3: (Laughs)....No! (also gesture behind himself) almost uh almost there are human...

13) S2: Uhhh? (tilts head to side in gesture of skepticism?)=

14) S3: =or my mood.

15) S1: Hmmm.

[

16) S2: I feel sometimes uh something uh something dangerous (feigns looking behind himself in fear)

17) S1: Do you feel (.) do you feel scare if (.) you go (.) go to (.) a haunted house?

18) S3: Ahh... Haunted house uhh ... human makes the house.

[

19) S2: Uhhh

20) S1: Yah?=

21) S3: =It isn't real ghost.

22) S1: But you fear it?

23) S3: Uhhh ... surprise Waaa! Ohhh! (indicates being startled by placing both hand over heart)

[

24) S2: Surprise?

25) S1: Uhhh (laughs) yeah yeah ok but ... you can see a picture (.) you can see a ghost in a picture (indicates shape of picture with thumb and index fingers)

26) S2: (laughter)

[

27) S3: Ohhh!

28) S1: How do you explain?

29) S3: Ohh... sunlight ... or the camera is bad =

[

30) S1: Yeah

31) S2: But on TV... uhh I watch people... who watch ghost

32) S1: Yeah. Yeah. Yeah.

[

33) S2: Talk about ghost

34) S3: Uhh (laughs) They are (inaudible) man uh... they want to the money and... they sell ( inaudible)

35) S1: So they are swindlers.


S2 and S3 demonstrated the ability to contribute to the progression of the conversation with appropriate responses and questions (active listening). They were able to communicate their intended meanings, using visual strategies to compensate where they lacked the English verbal ability (see lines 10, 12, 16 and 23). Their performance, based on the scales, warrants a B grading. S3 was given an "A" based on his demonstrated ability to overcome communication breakdowns (line 35), more effective elicitation of input (lines 17, 22 and 28), and the introduction of new points and topics (lines 1, 17, 25). As for English skills both S2 and S3 were given a "B" assessment based primarily on the demonstration of the necessary lexical knowledge and minimum grammatical knowledge (i.e. word order) to both encode and decode the necessary meanings. S1 was given a slightly higher grade,"B" primarily for greater demonstrated lexical knowledge (i.e. line 35) and the ability to construct some more complex utterances with reasonable standards of correctness ( a well constructed if clause in line 17). Crucial to the assessment of content is a judgement on what constitutes a topic of "relative complexity for false beginner". It is fair to say that a debate on the existence of ghosts constitutes a very complex topic choice indeed. Each of the students contributed arguments and disagreements in the development of the debate and thus warranted an "A" assessment.

The second transcript included is also an excerpt from an oral test under the heading of superstitions.

1) S1: How about you...what do you...believe (.) superstitions? Any superstitions?

2) S2: Uh. yes uhm... I believe partly... partly I believe it uhmm

3) S1: For example?

4) S2: For example uhmm uhm ...in my childhood my mother said to me uhmmm ... not not play with fire=

5) S1: =(inaudible)

6) S2: Or not play with fire uhh or... we will you will wet the bed ... next morning

7) S1: Next morning (laughs)

8) S2: Yes I uh I put it to the proof uhh try it uhh try it

9) S1: Try it=

10) S2: =Try it uh I wetted the bed next morning

11) S1: Really! (laugh)

12) S2: Yes ohh it was ... it was certain

13) S1: It was certain=

14) S2: =Certain I believe

15) S1: How about you?

16) S3: The superstition that... I believe is... has scientific base.

17) S1: Uh

18) S3: For example the western cloud is... glowing with the setting sun... it is clear up tomorrow.

19) S1: Oh=

20) S3: =clouds is move... west ahh.. from west to...

21) S2: From west to east?

[

22) S3: East.

23) S2: From west to east ... yes I see

[

24) S3: Uhhh

In this transcript S1 plays a critical role in facilitating student input (lines 1, 3 and 15) He also displays attentive listening skills primarily by echoing (lines 7, 9 and 13). There are no samples of S1 providing input thus it is difficult to assess "the ability to communicate most intended meanings". Given evidence of elicitation skills warranting an "A" grading and active listening warranting "B" grading, an AB grading would seem an appropriate compromise. In the first half of the conversation S2 demonstrates the ability to convey his intended meaning while in the second half he demonstrates the ability to overcome communication breakdowns (lines 21 and 23). Assuming that S2 would be able to demonstrate the ability to elicit partner input in other non-transcribed parts of the test (which was indeed the case) he would warrant an "A" grading under communication skills. S3 plays a primarily passive role in the conversation, contributing only when called on in line 16 (assuming the absence of visual input not evident in a simple transcript which was indeed the case). His input was halting and somewhat difficult to interpret, as indicated by the necessary input from S2, although ultimately decipherable. Given partial credit for being able to communicate his ideas he would warrant a "C" grading under communication skills.

Under English skills S1, primarily in the first half of the conversation, demonstrates the ability to interpret basic meanings (B level) as well as the ability to formulate questions (C level). In the absence of more input a BC grading is the best compromise. S2 demonstrates knowledge of some relatively complex lexis (play with fire, put it to the proof, wet the bed) as well as an expression ("it was certain") which, while understandable in context, should be seen as problematic. He also successfully completes a partner's attempted utterance (line 20) demonstrating the ability to dissemble, in real time, relatively complex language. He was given an AB grading. The utterances of S3, while comprehensible in context, can only be seen as problematic. A repeated, and relatively basic mistake, is the repeated use of "is" as if corresponding to the subject marker "wa" in Japanese. Granted the fact that he did demonstrate knowledge of the necessary lexis to communicate his intended meaning (albeit with some prompting in line 21) a BC grading would seem to be most fair.

Finally we come to content. Within this transcript S1 never had the opportunity to contribute, electing instead to play the role of facilitator. In this case he can only be assessed based on the complexity of his responses. Given him some credit for having demonstrated "some comprehension of the topic and ongoing discourse" and for having prepared questions relevant to the topic he warrants a BC grading. S2's anecdote of "putting a superstition to the proof" demonstrates unusual complexity and detail for a class consisting primarily of "false beginners" and warrants an A grading. S3's explanation of the scientific basis to a superstition is also of unusual complexity. Since it is the attempt, not the relative success, that is assessed he also warrants an A grading.

Conclusion


Any attempt to grade oral communicative competence must deal with the multi-faceted complexity of the area. The application of relative rankings in the form of rating scales must also deal with the additional complication of subjective evaluations of limited samples of performance. The grading process itself is dynamic, involving elicited performances, relevant rating scales, and subjective interpretations of both. Explicitly outlining the decision making process involved should help teachers to make explicit their own assumptions regarding communicative competence. More generally it should help to shed light on the actual basis for rating scales assessment, a crucial step towards true estimates of the validity of test scores.

- Return to the main article -

Newsletter: Topic IndexAuthor IndexTitle IndexDate Index
Main Page Background Links Network Join