Skip to main content
JALT Testing & Evaluation SIG

Main navigation

  • Home
  • About TEVAL SIG
    • TEVAL's Mission
    • Officers
    • Constitution
  • Publications
    • Shiken Journal
    • Shiken Current Issue
    • Statistics Corner: Questions and Answers about Language Testing Statistics (book)
    • Shiken Back Issues
    • Submission Guidelines
    • TEVAL News
    • Shiken Issues Pre May 2012
  • News
    • TEVAL Talk Time
    • Grants Information
    • Introduction to jMetrik
    • Shiken Journal Call for Papers
    • Statistics Corner
    • Ethical Issues in Language Testing - Dr. Isbell
    • Dr. Isbell Pansig Plenary Slides
  • Events
  • Join
  • Contact Us
    • Facebook
    • email
User account menu
  • Log in

Breadcrumb

  1. Home

A speaking comparison of text analysis tools: Levels of agreement and disagreement

Article appearing in Shiken 28.1 (Nov. 2024) pp. 38-55;

Article Doi: https://doi.org/10.37546/JALTSIG.TEVAL28.1-3

By Mart Christine Johnston
Takushoku University

Abstract

This paper explores agreement levels among text analysis tools and factors influencing text difficulty using 75 Grade 2 EIKEN texts. EIKEN is a Japanese proficiency test for high school graduates. Text analysis included word count, average sentence length, CEFR, CEFR-J levels (determined by Text Inspector and CVLA, respectively), Flesch Reading Ease Score, Flesch-Kincaid Grade Level, Lexile score, and total coverage (from AntWordProfiler) using the New General Service List (NGSL). Average sentence lengths had stronger correlations with other indices than word counts. Flesch-Kincaid Grade Level and Lexile reading levels correlated moderately due to reliance on sentence length. According to both tools, the texts are generally deemed appropriate for grades 8-9 in the US education system. Considering that text levels for second language classrooms are typically several levels lower than those in the US education system and recognizing that EIKEN Grade 2 is intended for high school graduates in Japan, it can be inferred that the texts align with expectations for high school graduates in Japan, equivalent to grades 8-9 in the US education system. CEFR and CEFR-J levels, although their text level assignments were similar, had only moderate correlations, reflecting metric differences between Text Inspector and CVLA. AntWordProfiler's total coverage showed weak correlations, focusing solely on word frequency. The results from this study show clear discrepancies in text difficulty depending on the type of measure used and call for varied approaches to text analysis.

Keywords: text analysis, EIKEN, text difficulty, text measures, Text Inspector, CVLA, AntWordProfiler, NGSL

Download full article (PDF)

PanSIG Conference, Chiba, May 16-18, 2025

PanSIG
Kanda University of International Studies

SHIKEN

A Journal of Language Testing and Evaluation in Japan

Site editors

  • Reset your password
RSS feed

JALT is the Japan Association for Language Teaching, a nonprofit organization dedicated to the improvement of language teaching and learning. The TEVAL SIG is a Special Interest Group of researchers within JALT who are interested in testing and the evaluation of language learning outcomes.

Powered by Drupal