Questions and answers about language testing statistics: Characteristics of sound qualitative research

Shiken:JALT Testing & Evaluation SIG Newsletter
Vol. 9 No. 2. Oct 2005. (p. 31 - 33) [ISSN 1881-5537]

Statistics Corner
Questions and answers about language testing statistics:

Characteristics of sound qualitative research

James Dean Brown
University of Hawai'i at Manoa

QUESTION: Many Shiken readers have never undertaken a qualitative research project, and they probably need to know something about triangulation. Could you briefly outline why triangulation is important, what the main types of triangulation are, and some of the most common triangulation procedures?

ANSWER: Certainly, let me begin with a definition of what I think research is. Then I will turn to the issues that qualitative researchers need to address in order in order to produce sound qualitative research. As I proceed through these explanations, you will see how triangulation fits into the whole picture.

What is Research?

In a survey that I conducted years ago for the TESOL organization (reported in Brown, 1992), I asked the respondents for their definitions of research. The results? A surprising variety of answers ranging from short, idealistic definitions (e.g., "Careful, thorough study" and "The search for the truth") to cynical ones (e.g., "Ignoring the obvious"). More recently in an article about applied linguistics research (Brown, 2004), I was able to settle on a single definition for research that was broad enough to include all the definitions listed in Brown (1992): research is "any systematic and principled inquiry."

How Can We Know If Qualitative Research is Systematic and Principled?

However, research can be systematic and principled in many different ways. Brown (2004) discusses those different ways in terms of what Newman and Benz (1998) called the qual-quant continuum. In general terms, good quantitative research (at one end of the qual-quant continuum) will be judged in terms of its reliability, validity, replicability, and generalizability, while sound qualitative research (at the other end of the continuum) will be judged in term of its dependability, credibility, confirmability, and transferability. Naturally, much of our research falls somewhere in between those two end points of the qual-quant continuum, or combines aspects of both. In this column, I will focus on the qualitative end of the continuum, and therefore on dependability, credibility, confirmability, and transferability

Dependability involves accounting for all the changing conditions in whatever is being studied as well as any changes in the design of the study that were needed to get a better understanding of the context. Dependability can be enhanced by using overlapping methods, stepwise replications, and/or inquiry audits (e.g., see Denzin, 1994, p. 513). Overlapping methods use carefully planned METHODOLOGICAL TRIANGULATION, or multiple data gathering procedures (e.g., observations, interviews, and questionnaires), in order to create overlapping (and therefore cross-validating) data. Stepwise replications involve TIME TRIANGULATION, that is, gathering data on multiple occasions (e.g., at the beginning, middle, and end of a school year), which helps in examining the consistency of the data and interpretations over time. Inquiry audits involve enlisting an outside expert "auditor" to verify the consistency of agreement among data, research methods, interpretations, conclusions, etc. Where appropriate, confidence in the dependability of a study can also be improved by doing quantitative analyses like intercoder/interrater agreement coefficients or other reliability estimates.

[ p. 31 ]

Credibility requires demonstrating, in one or more ways, that the research was designed to maximize the accuracy of identifying and describing whatever is being studied, especially as judged by the groups of people being studied. Credibility can be enhanced by using one or more of the following strategies: prolonged engagement, persistent observation, triangulation, peer debriefing, negative case analysis, and/or member checking (see Denzin, 1994, p. 513).

Prolonged engagement involves investing sufficient time and persistent observation involves using adequate numbers of observations, meetings, interviews, etc. so that participants feel enough confidence and trust in the researcher to allow for adequate study of the cultural context and adequate checks for misinformation (Davis, 1992, p. 606; 1995, p. 445).

Various forms of triangulation may also enhance credibility: SOURCE TRIANGULATION involves gathering data from multiple sources (e.g., people in different roles, like students, teachers, and administrators) in order to minimize and understand any differences/biases held by people in various roles. INVESTIGATOR TRIANGULATION involves using multiple researchers to interpret the data in order to minimize and understand any differences/biases the researchers may have. LOCATION TRIANGULATION involves gathering data at multiple sites (e.g., three different schools) in order to minimize and understand any differences/biases that might be introduced by the participants in each of the institutions.

Credibility can also be enhanced by using peer debriefing (i.e., critical examination and evaluation by a qualified outside researcher of the study design, data collection, analyses, etc.); negative case analysis (i.e., intentionally searching for and analyzing examples of data or participants that contradict the overall interpretations in a study); and member checking (i.e., verifying the researcher's interpretations and conclusions with the various groups of participants themselves).

Confirmability entails full revelation of the data upon which all interpretations are based, or at least the availability of the data for inspection. In other words, the reader of the research report should be able to examine the data to confirm the results/interpretations. Confirmability is sometimes enhanced by using audit trails (i.e., a "residue of records stemming from inquiry", Lincoln & Guba, 1985, p. 319). According to Denzin (1994, p. 513), "confirmability builds on audit trails...and involves the use of written field notes, memos, a field diary, process and personal notes, and a reflexive journal." Clearly, thorough record keeping and preservation of data for potential inspection are crucial to this strategy. Some researchers will append their data (including transcripts, instructions, etc.) to their report, or at least include crucial examples for inspection by the reader. Naturally, if the reader can inspect the data, the interpretations and results will be maximally confirmable.

Transferability involves demonstrating the applicability of the results of the study in one context to other contexts. Transferability can be enhanced by providing what is often referred to as thick description (i.e., giving enough detail so the readers can decide for themselves if the results are transferable to their own contexts). Thick description also "involves an emic perspective, which demands description that includes the actors' interpretations and other social and/or cultural information" (Davis, 1995, p. 434). Marshall and Rossman (1989, p. 145) note that transferability is the responsibility of the person seeking to apply the results of the study to a new context, that is, it is the responsibility of the reader. "In this way, the responsibility of the original investigator ends in providing sufficient descriptive data to make such similarity judgments possible" (Davis 1992, p. 606).

[ p. 32 ]

Conclusion

In direct answer to your original question, triangulation is a key concept in qualitative research. Variants of triangulation are used to enhance both the dependability (i.e., methodological and time triangulation) and credibility (i.e., source, investigator, and location triangulation) of such research. Other types of triangulation exist (e.g., theory triangulation and interdisciplinary triangulation) that serve additional purposes (see Brown 2001, pp. 227-231). However the issue of triangulation is not that simple: as Fielding and Fielding (1986) point out, "the important feature of triangulation is not the simple combination of different kinds of data, but the attempt to relate them so as to counteract the threats to validity identified in each" (p. 31). Hence, to be used effectively, triangulation must be carefully thought through and planned.

Those who find qualitative research methods intriguing may find some or all of the following general qualitative research books useful: Denzin and Lincoln (1994), Fetterman (1989), Glesne (1998), Huberman & Miles (2002), Marshall and Rossman (1989), Maxwell (2004), and Miles and Huberman (1984). Five other books provide at least a chapter length treatment of qualitative language research methods: Brown (2001), and Brown and Rodgers (2002), Freeman (1998), Johnson (1992), and Nunan (1992).

References

Brown, J. D. (1992). What is research? TESOL Matters, 2 (5), 10.

Brown, J. D. (2001). Using surveys in language programs. Cambridge: Cambridge University Press.

Brown, J. D. (2004). Research methods for Applied Linguistics: Scope, characteristics, and standards. In A. Davies & C. Elder (Eds.), The handbook of applied linguistics (pp. 476-500). Oxford: Blackwell.

Brown, J. D., & Rodgers, T. (2002). Doing applied linguistics research. Oxford: Oxford University Press.

Davis, K. A. (1992). Validity and reliability in qualitative research on second language acquisition and teaching: Another researcher comments.... TESOL Quarterly, 26, 605-608.

Davis, K. A. (1995). Qualitative theory and methods in applied linguistics research. TESOL Quarterly, 29, 427-453.

Denzin, N. K. (1994). The art and politics of interpretation. In N. K. Denzin & Y. S. Lincoln (Eds.), Handbook of qualitative research (pp. 500-515). Thousand Oaks, CA: Sage.

Denzin, N. K., & Lincoln, Y. S. (Eds.). (1994). Handbook of qualitative research. Thousand Oaks, CA: Sage.

Fetterman, D. M. (1989). Ethnography: Step by step. Newbury Park, CA: Sage.

Fielding, N. G., & Fielding, J. L. (1986). Linking data. Beverly Hills, CA: Sage.

Freeman, D. (1998). Doing teacher research: From inquiry to understanding. Boston, MA: Heinle & Heinle.

Glesne, C. (1998). Becoming qualitative researchers: An introduction (2nd ed.). London: Allyn & Bacon.

Huberman, M., & Miles, M. B. (Eds.). (2002). The qualitative researcher's companion: Classic and contemporary readings. Beverly Hills, CA: Sage.

Johnson, D. M. (1992). Approaches to research in second language learning. New York: Longman.

Lincoln, Y. & Guba, E. (1985). Naturalistic inquiry. Beverly Hills, CA: Sage.

Marshall, C., & Rossman, G. B. (1989). Designing qualitative research. Newbury Park, CA: Sage.

Maxwell, J. A. (2004). Qualitative research design: An interactive approach (2nd ed.). Beverly Hills, CA: Sage.

Miles, M. B., & Huberman, A. M. (1984). Qualitative data analysis: A sourcebook of new methods. Beverly Hills, CA: Sage.

Newman, I., & Benz, C. R. (1998). Qualitative-quantitative research methodology: Exploring the interactive continuum. Carbondale, IL: Southern Illinois University Press.

Nunan, D. (1992). Research methods in language learning. Cambridge: Cambridge University Press.

Where to Submit Questions:

Please submit questions for this column to the following e-mail or snail-mail addresses:

brownj@hawaii.edu

JD Brown, Department of Second Language Studies
University of Hawai'i at Manoa
1890 East-West Road, Honolulu, HI 96822 USA

[ p. 33 ]

NEWSLETTER: Topic Index

Author Index

Title Index

Date Index
TEVAL SIG: Main Page

Background

Links

Network

Join

STATISTICS CORNER ARTICLES:
#1 #2 #3 #4 #5 #6 #7 #8 #9 #10 #11 #12 #13 #14 #15 #16 #17 #18 #19 #20 #21 #22 #23 #24 #25 #25 #26 #27 #28 #29 #30 #31 #32 #33 #34