Title: ANALYSIS OF DISCOURSE FEATURES AND VERIFICATION OF SCORING LEVELS FOR INDEPENDENT AND INTEGRATED PROTOTYPE WRITTEN TASKS FOR THE NEW TOEFL®
Abstract: ETS Research Report SeriesVolume 2005, Issue 1 p. i-77 ArticleFree Access ANALYSIS OF DISCOURSE FEATURES AND VERIFICATION OF SCORING LEVELS FOR INDEPENDENT AND INTEGRATED PROTOTYPE WRITTEN TASKS FOR THE NEW TOEFL® Alister Cumming, Alister Cumming Ontario Institute for Studies in Education of the University of Toronto, CanadaSearch for more papers by this authorRobert Kantor, Robert Kantor ETS, Princeton, NJSearch for more papers by this authorKyoko Baba, Kyoko Baba Ontario Institute for Studies in Education, University of Toronto, CanadaSearch for more papers by this authorKeanre Eouanzoui, Keanre Eouanzoui Ontario Institute for Studies in Education, University of Toronto, CanadaSearch for more papers by this authorUsman Erdosy, Usman Erdosy Ontario Institute for Studies in Education, University of Toronto, CanadaSearch for more papers by this authorMark Jamse, Mark Jamse Ontario Institute for Studies in Education, University of Toronto, CanadaSearch for more papers by this author Alister Cumming, Alister Cumming Ontario Institute for Studies in Education of the University of Toronto, CanadaSearch for more papers by this authorRobert Kantor, Robert Kantor ETS, Princeton, NJSearch for more papers by this authorKyoko Baba, Kyoko Baba Ontario Institute for Studies in Education, University of Toronto, CanadaSearch for more papers by this authorKeanre Eouanzoui, Keanre Eouanzoui Ontario Institute for Studies in Education, University of Toronto, CanadaSearch for more papers by this authorUsman Erdosy, Usman Erdosy Ontario Institute for Studies in Education, University of Toronto, CanadaSearch for more papers by this authorMark Jamse, Mark Jamse Ontario Institute for Studies in Education, University of Toronto, CanadaSearch for more papers by this author First published: 08 August 2014 https://doi.org/10.1002/j.2333-8504.2005.tb01990.xCitations: 11AboutPDF ToolsRequest permissionExport citationAdd to favoritesTrack citation ShareShare Give accessShare full text accessShare full-text accessPlease review our Terms and Conditions of Use and check box below to share full-text version of article.I have read and accept the Wiley Online Library Terms and Conditions of UseShareable LinkUse the link below to share a full-text version of this article with your friends and colleagues. Learn more.Copy URL Share a linkShare onFacebookTwitterLinkedInRedditWechat ABSTRACT We assessed whether and how the discourse written for prototype integrated tasks (involving writing in response to print or audio source texts) field tested for the new TOEFL® differs from the discourse written for independent essays (i.e., the TOEFL essay). We selected 216 compositions written for 6 tasks by 36 examinees in a field test—representing Score Levels 3, 4, and 5 on the TOEFL essay—then coded the texts for lexical and syntactic complexity, grammatical accuracy, argument structure, orientations to evidence, and verbatim uses of source text. Analyses with nonparametric MANOVAs, following a 3-by-3 (task type by English proficiency level) within-subjects factorial design, showed that the discourse produced for the integrated writing tasks differed significantly at the lexical, syntactic, rhetorical, and pragmatic levels from the discourse produced in the independent essay on most of these variables. In certain analyses, these differences were also obtained across the 3 ESL proficiency levels. REFERENCES Abdi, R. (2002). Interpersonal metadiscourse: An indicator of interaction and identity. Discourse Studies, 4, 139– 145. Anderson, M. J. (1999). Non-parametric MANOVA [Computer software]. Sydney, Australia: University of Sydney, Centre for Research on Ecological Impacts of Coastal Cities. Anderson, M. J. (2001). A new method for non-parametric multivariate analysis of variance. Austral Ecology, 26, 32– 46. Bardovi-Harlig, K., & Bofman, T. (1989). Attainment of syntactic and morphological accuracy by advanced language learners. Studies in Second Language Acquisition, 11, 17– 34. Barton, E. L. (1993). Evidentials, argumentation, and epistemological stance. College English, 55, 745– 769. Biber, D. (1986). Spoken and written textual dimensions in English: Resolving the contradictory findings. Language, 62, 384– 411. Biber, D. (1988). Variation across speech and writing. Cambridge, UK: Cambridge University Press. Biber, D. (1995). Dimensions of register variation: A corpus-linguistic comparison. Cambridge, UK: Cambridge University Press. Biber, D., & Finegan, E. (1989). Styles of stance in English: Lexical and grammatical marking of evidentiality and affect. Text, 9, 93– 124. Britt, M. & Aglinskas, C. (2002). Improving students' ability to identify and use source information. Cognition and Instruction, 20, 485– 522. Brown, A., & Day, J. D. (1983). Macrorules for summarizing texts: The development of expertise. Journal of Verbal Learning, 22, 1– 14. Carlisle, R. S. (1989). The writing of Anglo and Hispanic elementary school students in bilingual, submersion, and regular programs. Studies in Second Language Acquisition, 11, 257– 280. Chenoweth, A., & Hayes, J. R. (2001). Fluency in writing: Generating text in L1 and L2. Written Communication, 18, 90– 98. Chipere, N., Malvern, D., Duran, P., & Richards, B. (2003, March). Some quantifiable aspects of literacy development. Paper presented at the annual meeting of the American Association of Applied Linguistics, Washington, DC. Cohen, J. (1969). Statistical power analysis for the behavioral sciences. New York: Academic Press. Connor, U. (1990). Linguistic/rhetorical measures for international persuasive student writing. Research in the Teaching of English, 24, 67– 87. Connor, U. (1991). Linguistic/rhetorical measures for evaluating ESL writing. In L. Hamp-Lyons (Ed.), Assessing second language writing in academic contexts (pp. 215– 225). Norwood, NJ: Ablex. Conrad, S., & Biber, D. (2000). Adverbial marking of stance in speech and writing. In S. Hunston & G. Thompson (Eds.), Evaluation in text: Authorial stance and the construction of discourse (pp. 56– 73). Oxford, UK: Oxford University Press. Crammond, J. (1998). The uses and complexity of argument structures in student persuasive writing. Written Communication, 15, 230– 268. Crismore, A., Markkanen, R., & Steffensen, M. (1993). Metadiscourse in persuasive writing: A study of texts written by American and Finnish university students. Written Communication, 10, 39– 71. Crowhurst, M., & Piche, G. (1979). Audience and mode of discourse effects on syntactic complexity in writing at two grade levels. Research in the Teaching of English, 13, 101– 109. Cumming, A. (2001). Learning to write in a second language: Two decades of research. International Journal of English Studies, 1, 1– 23. Cumming, A. (2002). Assessing L2 writing: Alternative constructs and ethical dilemmas. Assessing Writing, 8, 73– 83. Cumming, A., Grant, L., Mulcahy-Ernt, P., & Powers, D. (2004). A teacher-verification study of prototype reading and speaking tasks for New TOEFL (TOEFL Monograph No. MS-26, ETS RM-04-05), Princeton, NJ: ETS. Cumming, A., Kantor, R., & Powers, D. (2001). Scoring TOEFL essays and TOEFL 2000 prototype tasks: An investigation into raters' decision making and development of a preliminary analytic framework (TOEFL Monograph No. MS-22, ETS RM-01-22). Princeton, NJ: ETS. Cumming, A., Kantor, R., & Powers, D. (2002). Decision making while scoring ESL/EFL compositions: A descriptive model. Modern Language Journal, 86, 67– 96. Cumming, A., Kantor, R., Powers, D., Santos, T., & Taylor, C. (2000). TOEFL 2000 writing framework: A working paper (TOEFL Monograph No. MS-18, ETS RM-00-05). Princeton, NJ: ETS. Cumming, A., & Mellow, D. (1996). An investigation into the validity of written indicators of second language proficiency. In A. Cumming & R. Berwick (Eds.), Validation in language testing (pp. 72– 93). Clevedon, UK: Multilingual Matters. Cumming, A., Rebuffot, J., & Ledwell, M. (1989). Reading and summarizing challenging texts in first and second languages. Reading and Writing, 2, 201– 219. Day, J. D. (1986). Teaching summarization skills: Influences of student ability level and strategy difficulty. Cognition and Instruction, 3, 193– 210. Deckert, G. (1993). Perspectives on plagiarism from ESL students in Hong Kong. Journal of Second Language Writing, 2, 131– 148. ETS. (2002). LanguEdge courseware: Handbook for scoring speaking and writing. Princeton, NJ: Author. Engber, C. (1995). The relationship of lexical proficiency to the quality of ESL compositions. Journal of Second Language Writing, 4, 139– 155. Faigley, L. (1979). The influence of generative rhetoric on the syntactic maturity and writing effectiveness of college freshmen. Research in the Teaching of English, 13, 197– 206. Frase, L., Faletti, J., Ginther, A., & Grant, L. (1999). Computer analysis of the TOEFL Test of Written English (TOEFL Research Rep. No. 64, ETS RR-98-42). Princeton, NJ: ETS. Grant, L., & Ginther, A. (2000). Using computer-tagged linguistic features to describe L2 writing differences. Journal of Second Language Writing, 9, 123– 145. Hamp-Lyons, L., & Henning, G. (1991). Communicative writing profiles: An investigation of the transferability of a multiple-trait scoring instrument across ESL writing assessment contexts. Language Learning, 41, 337– 373. Hamp-Lyons, L., & Kroll, B. (1996). TOEFL 2000. Writing: Composition, community, and assessment (TOEFL Monograph Series Rep. No. 5, ETS RM-96-05). Princeton, NJ: ETS. Haswell, R. H. (1988). Error and change in college student writing. Written Communication, 5, 479– 499. Homburg, T. J. (1984). Holistic evaluation of ESL compositions: Can it be validated objectively? TESOL Quarterly, 18, 87– 107. Howard, R. (1995). Plagiarisms, authorships and the academic death penalty. College English, 57, 788– 806. Hunt, K. W. (1965). Grammatical structures at three grade levels (NCTE Research Rep. No. 3). Urbana, IL: The National Council of Teachers of English. Hyland, K. (1996). Talking to the academy: Forms of hedging in science research articles. Written Communication, 13, 251– 281. Ishikawa, S. (1995). Objective measurement of low-proficiency EFL narrative writing. Journal of Second Language Writing, 4, 51– 69. Ivanic, R., & Camps, D. (2001). I am how I sound: Voice as self-representation in L2 writing. Journal of Second Language Writing, 10, 3– 33. Jamieson, J., Jones, S., Kirsch, I., Mosenthal, P., & Taylor, C. (2000). TOEFL 2000 framework: A working paper (TOEFL Monograph No. MS-16, RM-00-03). Princeton, NJ: ETS. Kintsch, E. (1990). Macroprocesses and microprocesses in the development of summarization skill. Cognition and instruction, 7, 161– 195. Kintsch, W., & van Dijk, T. (1978). Toward a model of text comprehension and production. Psychological Review, 85, 363– 394. Knudson, R. (1992). Analysis of argumentative writing at two grade levels. Journal of Educational Research, 85, 169– 179. Kroll, B. (1988). How college freshmen view plagiarism. Written Communication, 5, 203– 221. Larsen-Freeman, D. (1978). An ESL index of development. TESOL Quarterly, 12, 439– 448. Larsen-Freeman, D., & Strom, V. (1977). The construction of a second language acquisition index of development. Language Learning, 27, 123– 134. Laufer, B. (1991). The development of L2 lexis in the expression of the advanced learner. Modern Language Journal, 75, 440– 448. Laufer, B., & Nation, P. (1995). Vocabulary size and use: Lexical richness in L2 written composition. Applied Linguistics, 16, 307– 322. Lee, W., Kantor, R., & Mollaun, P. (2002, April). Score reliability as an essential prerequisite for validating new writing and speaking tasks for TOEFL. Paper presented at the annual TESOL Convention, Salt Lake City, UT. Legendre, P., & Anderson, M. J. (1999). Distance-based redundancy analysis: Testing multispecies responses in multifactorial ecological experiments. Ecology Monographs, 69, 1– 24. Legendre, P., & Legendre, L. (1998). Numerical ecology ( 2nd English ed.). Amsterdam: Elsevier. Lennon, P. (1991). Error: Some problems of definition, identification, and distinction. Applied Linguistics, 12, 180– 196. Levine, T. R., & Hullett, C. R. (2002). Eta squared, partial eta squared, and misreporting of effect size in communication research. Human Communication Research, 28, 612– 625. McArdle, B., & Anderson, M. J. (2001). Fitting multivariate models to community data: A comment on distance-based redundancy analysis. Ecology, 82, 290– 297. McCann, T. (1989). Student argumentative writing knowledge and ability at three grade levels. Research in the Teaching of English, 23, 62– 76. McCarthy Young, K., & Leinhardt, G. (1998). Writing from primary documents: A way of knowing in history. Written Communication, 15, 25– 68. Neter, J., Wasserman, W., & Kutner, M. H. (1985). Applied linear statistical models ( 2nd ed.). Homewood, IL: Richard D. Irwin, Inc. Pennycook, A. (1996). Borrowing others' words: Text, ownership, memory, and plagiarism. TESOL Quarterly, 30, 201– 230. Perkins, K. (1980). Using objective methods of attained writing proficiency to discriminate among holistic evaluations. TESOL Quarterly, 14, 61– 69. Plungian, V. (2001). The place of evidentiality within the universal grammatical space. Journal of Pragmatics, 33, 349– 357. Polio, C. (1997). Measures of linguistic accuracy in second language writing research. Language Learning, 47, 101– 143. Quirk, R., Greenbaum, S., Leech, G., & Svartvik, J. (1985). A comprehensive grammar of the English language. London: Longman. Read, J. (2000). Assessing vocabulary. Cambridge, UK: Cambridge University Press. Reid, J. (1986). Using the Writer's Workbench in composition teaching and testing. In C. Stansfield (Ed.), Technology and language testing (pp. 167– 188). Alexandria, VA: TESOL. Reppen, R. (1994). Variation in elementary student language: A multi-dimensional perspective. Unpublished doctoral dissertation, Northern Arizona University, Flagstaff. Rifkin, B., & Roberts, F. (1995). Error gravity: A critical review of research design. Language Learning, 45, 511– 537. Rosenfeld, M., Leung, S., & Oltman, P. K. (2001). The reading, writing, speaking, and listening tasks important for academic success at the undergraduate and graduate levels (TOEFL Monograph No. MS-21, ETS RM-01-03). Princeton, NJ: ETS. Rouet, J., Favart, M., Gaonach, D., & Lacroix, N. (1996). Writing from multiple documents: Argumentation strategies in novice and expert history students. In G. Rijlaarsdam, H. Van den Bergh, & M. Couzija (Eds.), Theories, models and methodology in writing research (pp. 44– 60). Amsterdam: University of Amsterdam Press. Scollon, R. (1994). As a matter of fact: The changing ideology of authorship and responsibility in discourse. World Englishes, 13, 33– 46. M. D. Shermis, & J. C. Burstein (Eds.). (2003). Automated essay scoring: A cross-disciplinary perspective. Mahway, NJ: Erlbaum. Shi, L. (2004). Textual borrowing in second-language writing. Written Communication, 21, 171– 200. Stansfield, C., & Ross, J. (1988). A long-term research agenda for the Test of Written English. Language Testing, 5, 160– 186. Stewart, M. F., & Grobe, C. H. (1979). Syntactic maturity, mechanics of writing, and teachers' quality ratings. Research in the Teaching of English, 13, 207– 215. Stromso, H. I., Braten, I., & Samuelstuen, M. S. (2003). Students' strategic use of multiple sources during expository text reading: A longitudinal think-aloud study. Cognition and Instruction, 21, 113– 147. Tabachnik, B. G., & Fidell, L. S. (1996). Using multivariate statistics ( 3rd ed.). New York: Harper Collins. Thompson, G. (1996). Voices in the text: Discourse perspectives on language reports. Applied Linguistics, 17, 501– 530. Thompson, G., & Yiyun, Y. (1991). Evaluation in the reporting verbs used in academic papers. Applied Linguistics, 12, 365– 382. Toulmin, S. E. (1958). The uses of argument. Cambridge, UK: Cambridge University Press. Toulmin, S., Rieke, R., & Janik, A. (1984). An introduction to reasoning ( 2nd ed.). New York: Macmillan. Way, D. P., Joiner, E. G., & Seaman, M. A. (2000). Writing in the secondary foreign language classroom: The effects of prompts and tasks on novice learners of French. The Modern Language Journal, 84, 171– 184. Wiley, J., & Voss, J. (1999). Constructing arguments from multiple sources: Tasks that promote understanding and not just memory for text. Journal of Educational Psychology, 91(2), 301– 311. Wineburg, S. S. (1994). Historical problem solving: A study of the cognitive processes used in the evaluation of documentary and pictorial evidence. Journal of Educational Psychology, 83, 73– 87. Winograd, P. N. (1984). Strategic difficulties in summarizing texts. Reading Research Quarterly, 19(4), 404– 425. Wolfe-Quintero, K., Inagaki, S., & Kim, H-Y. (1998). Second language development in writing: Measures of fluency, accuracy & complexity. Honolulu, HI: University of Hawaii at Manoa. Yeh, S. (1998). Empowering education: Teaching argumentative writing to cultural minority middle-school students. Research in the Teaching of English, 33, 49– 83. Citing Literature Volume2005, Issue1June 2005Pages i-77 ReferencesRelatedInformation
Publication Year: 2005
Publication Date: 2005-06-01
Language: en
Type: article
Indexed In: ['crossref']
Access and Citation
Cited By Count: 43
AI Researcher Chatbot
Get quick answers to your questions about the article from our AI researcher chatbot