Cognitive Validity: A Review of its Rationale, Definition, Models and Empirical Studies

doi:10.3969/j.issn.1674-8921.2020.06.008

Abstract

Abstract:

Cognitive validity, as an important aspect of Weir's socio-cognitive validation framework (2005), refers to the design of test tasks which elicit from test-takers cognitive processes similar to those required by language users in their real-life communicative domains. Weir's framework not only allows for a better understanding of the constructs measured in language tests, but also complements traditional construct validation studies which feature factor analysis in the production of validity evidence. This paper offers a careful review on cognitive validity in terms of its theoretical foundation, concept development, cognitive models and empirical studies. The paper also indicates certain limitations in relevant studies, in particular concerning their model use, scope, methodology and application of the findings. It is to be hoped that this short review will be of value for researchers in their exploration of cognitive validity, thus contributing more to test design and use, and to the whole language assessment validation process.

Key words: language assessment, cognitive validity, cognitive processes, construct, validation

CLC Number:

XU Wandong, GU Xiangdong. Cognitive Validity: A Review of its Rationale, Definition, Models and Empirical Studies[J]. Contemporary Foreign Languages Studies, 2020, 20(6): 68-78.

References 65

[1]	Alderson J. C. 2000. Assessing Reading[M]. Cambridge: Cambridge University Press.
[2]	Anderson N. J., L. Bachman, K. Perkins & A. Cohen. 1991. An exploratory study into the construct validity of a reading comprehension test: Triangulation of data sources[J]. Language Testing 8(1):41-66.
[3]	Aryadoust, V. & L. M. Zhang. 2015. Fitting the mixed Rasch model to a reading comprehension test: Exploring individual difference profiles in L2 reading[J]. Language Testing 33(4):529-553.
[4]	Bachman L. F. 1990. Fundamental Considerations in Language Testing[M]. Oxford: Oxford University Press.
[5]	Bachman L. F. & A. S. Palmer. 2010. Language Assessment in Practice: Developing Language Assessments and Justifying Their Use in the Real World[M]. Oxford: Oxford University Press.
[6]	Bax S. 2013. The cognitive processing of candidates during reading tests: Evidence from eye-tracking[J]. Language Testing 30(4):441-465.
[7]	Bax, S. & C. J. Weir. 2012. Investigating learners' cognitive processes during a computer-based CAE reading test [A]. H.Khalifa & F. Barker (eds.). Cambridge ESOL: Research Notes[C]. Cambridge: Océ Ltd. 3-14.
[8]	Baxter G. P. & R. Glaser. 1998. Investigating the cognitive complexity of science assessments [J]. Educational Measurement: Issues and Practices 17(3):37-45.
[9]	Carroll J. B. 1968. The psychology of language testing[A]. A. Davies (ed.). Language Testing Symposium: A Psycholinguistic Approach[C]. Oxford: Oxford University Press. 46-69.
[10]	Cohen A. D. 2000. Exploring strategies in test-taking:Fine-tuning verbal reports from respondents[A]. In G.Ekbatani & P.Pierson (eds.). Learner-directed Assessment in ESL[M]. Mahwah: Lawrence Erlbaum Associates. 127-150.
[11]	Cohen A. D. 2006. The coming of age of research on test-taking strategies[J]. Language Assessment Quarterly 3( 4): 307-331.
[12]	Cohen A. D. 2011. Strategies in Learning and Using a Second Language[M]. New York: Pearson.
[13]	Cohen A. D. 2012. Test-taking strategies and task design[A]. G.Fulcher & F. Davidson (eds.). The Routledge Handbook of Language Testing[C]. New York: Routledge. 262-277.
[14]	Cohen A. D. & A. T. Upton. 2006. Strategies in responding to the new TOEFL reading tasks[J/OL]. ETS Research Report Series (1):1-127. [2019-03-01].https://onlinelibrary.wiley.com/doi/epdf/10.1002/j.2333-8504.2006.
[15]	Cohen A. D. & A. T. Upton. 2007. I want to go back to the text: Response strategies on the reading subtest of the new TOEFL[J]. Language Testing 24(2): 209-250.
[16]	Culter, A. & C.Clifton. 1999. Comprehending spoken language: A blueprint of the listener[A]. & P.Hagoort (eds.).The Neurocogition of Language[C]. Oxford: Oxford University Press.123-155.
[17]	Davies A., A. Brown C. Elder K. Hill T. Lumley & T. McNamara. 1999. Dictionary of Language Testing[M]. Cambridge: Cambridge University Press.
[18]	Field J. 2004. Psycholinguistics: The Key Concepts[M]. London: Routledge.
[19]	Field J. 2011. Cognitive validity[A]. L. Taylor (ed.). Examining Speaking: Research and Practice in Assessing Second Language Listening[C]. Cambridge: Cambridge University Press. 65-111.
[20]	Field J. 2012. The cognitive validity of the lecture-based question in the IELTS Academic Listening paper[A]. L.Taylor & C. J. Weir (eds.). IELTS Collected Papers 2: Research in Reading and Listening Assessment[C]. Cambridge: Cambridge University Press. 391-453.
[21]	Field J. 2013. Cognitive validity[A]. A.Geranpayeh & L. Taylor (eds.). Examining Listening: Research and Practice in Assessing Second Language Listening[C]. Cambridge: Cambridge University Press. 77-151.
[22]	Gorin J. S. 2006. Test design with cognition in mind[J]. Educational Measurement: Issues and Practices 42: 351-373.
[23]	Glaser R. 1991. Expertise and assessment[A]. M. C.Wittrock & E. L. Baker (eds.). Testing and Cognition[C]. New Jersey: Englewood Cliffs. 17-30.
[24]	Gao, X. & X. Gu. 2008. An introspective study on test-taking process for banked cloze[J]. CELEA Journal 31(4) : 3-16.
[25]	Grabe W. & R. B. Kaplan. 1996. Theory and Practice of Writing: An Applied Linguistic Perspective[M]. London: Longman.
[26]	Geranpayeh A. & L. Taylor. 2013. Examining Listening: Research and Practice in Assessing Second Language Listening[M]. Cambridge: Cambridge University Press.
[27]	Hayes J. R. & L. S. Flower. 1980. Identifying the organisation of writing process [A]. L. M.Gregg & E. R. Steinberg (eds.). Cognitive Processes in Writing[C]. Hillsdale: Lawrence Erlbaum Associates. 3-30.
[28]	Karabenick A. A., M. E. Woolley, J. M. Friedel, B. V. Ammon, J. Blazevski, C. R. Bonney, E. D. Groot, A. C. Gilbert, L. Musu, T. M. Kempler & K. Kelly. 2007. Cognitive processing of self-report items in educational research: Do they think what we mean?[J]. Educational Psychologist 42: 139-151.
[29]	Kellogg R. T. 1994. The Psychology of Writing[M]. Cambridge: Oxford University Press.
[30]	Kellogg R. T. 1996. A model of working memory in writing[A]. C. M.Levy & S. Ransdell (eds.). The Science of Writing[C]. Mahwah: Lawrence Erlbaum. 57-72.
[31]	Khalifa H. & C. J. Weir. 2009. Examining Reading: Research and Practice in Assessing Second Language Reading[M]. Cambridge: Cambridge University Press.
[32]	Lado R. 1961. Language Testing[M]. New York: McGraw-Hill.
[33]	Levelt W. 1989. Speaking: From Intention to Articulation[M]. Cambridge: MIT Press.
[34]	Levelt W. 1999. Language production: A blueprint of the speaker[A]. C.Brown & P. Hagoort (eds.). Neurocognition of Language[C]. Oxford: Oxford University Press. 83-122.
[35]	Linn R., E. L. Baker & S. B. Dunbar. 1991. Complex, performance-based assessment: Expectations and validation criteria[J]. Educational Researcher 20(8): 15-21.
[36]	Leighton J. P. & M. J. Gierl. 2007. Defining and evaluating models of cognition used in educational measurement to make inferences about examinees' thinking processes [J]. Educational Measurement: Issues and Practices 26(2): 3-16.
[37]	McCray, G. & T. Brunfaut. 2018. Investigating the construct measured by banked gap-fill items: Evidence from eye-tracking[J]. Language Testing 35(1): 51-73.
[38]	Messick S. 1989. Validity[A]. R. Linn (ed.). Educational Measurement[C]. Washington: American Council on Education. 13-103.
[39]	Newton P. E. & S. D. Shaw. 2014. Validity in Educational & Psychological Assessment[M]. London: SAGE Publication Inc.
[40]	Owen N. 2016. An Evidence-centred Approach to Reverse Engineering: Comparative Analysis of IELTS and TOEFL iBT Reading Sections[D]. University of Leicester.
[41]	O'Sullivan, B. & C. J. Weir. 2011. Test development and validation [A]. B. O'Sullivan (ed.). Language Testing: Theories and Practices[C]. Basingstoke: Palgreve Macmillan. 13-32.
[42]	Ruiz-Primo M. A., R. J. Shavelson, M. Li & S. E. Schultz. 2001. On the validity of cognitive interpretations of scores from alternative concept-mapping techniques[J]. Educational Assessment 7(2):99-141.
[43]	Smith M. D. 2017. Cognitive validity: Can multiple-choice items tap historical thinking processes[J]. American Educational Research Journal 54(6):1256-1287.
[44]	Storey P. 1997. Examining the test-taking process: A cognitive perspective on the discourse cloze test[J]. Language Testing 14(2):214-231.
[45]	Snow R. E., L. Corno & D. Jackson. 1996. Individual differences in affective and conative functions [A]. D. C.Berliner & R. C. Calfee (eds.). Handbook of Educational Psychology[C]. New York: Macmillan. 243-310.
[46]	Shaw S. D. & C. J. Weir. 2007. Examining Writing: Research and Practice in Assessing Second Language Writing[M]. Cambridge: Cambridge University Press.
[47]	Taylor L. 2011. Examining Speaking: Research and Practice in Assessing second Language Speaking[M]. Cambridge: Cambridge University Press.
[48]	Thelk A. D. & E. R. Hoole. 2006. What are you thinking? Postsecondary student think-alouds of scientific and quantitative reasoning items [J]. The Journal of General Education 55(1):17-39.
[49]	Thompson, I. & J. Rubin. 1996. Can strategy instruction improve listening comprehension?[J]. Foreign Language Annals 29( 3): 331-342.
[50]	Urquhart A. H. & C. J. Weir. 1998. Reading in a Second Language: Process, Product and Practice[M]. London: Longman.
[51]	VanderVeen. A., K. Huff, M. Gierl, D. S. McNamara, M. M. Louwerse & A. C. Graesser. 2006. Developing and validating instructionally relevant reading competency profiles measured by the critical reading section of the SAT [A]. In D. S. McNamara (ed.). Reading Comprehension Strategies:Theory, Intervention and Technologies[C]. Mahwah: Lawrence Erlbaum. 137-174.
[52]	Weir C. J. 2005. Language Testing and Validation: An Evidence-based Approach[M]. Basingstoke: Palgrave Macmillan.
[53]	Weir C. J. 1988. Construct validity[A]. A.Hughes, D. & C. J. Weir (eds.). ELT Validation Project: Proceeding of a Conference Held to Consider the ELTS Validation Project Report[C]. Cambridge: The University of Cambridge Local Examination Syndicate. 38-65.
[54]	Weir, C. J., R. Hawkey, T. Green & S. Devi. 2009. The cognitive processes underlying the academic reading construct as measured by IELTS [A]. In P. Thompson (ed.). International English Language Testing System (IELTS) Research Reports 2009 [C]. Canberra: British Council and IELTS Australia. 157-189.
[55]	Xi, X. & Y. Sawaki. 2017. Methods of test validation [A]. E.Shohamy, L. G.Or & M.Stephen (eds.). Language Testing and Assessment (3^rd edition)[C]. New York: Springer. 170-186.
[56]	Yamashita J. 2003. Processes of taking a gap-filling test: Comparison of skilled and less skilled EFL readers[J]. Language Testing 20(3): 267-293.
[57]	Yang, X. & S. E. Embretson. 2007. Construct validity and cognitive diagnostic assessment [A]. J. P.Leighton & M. J. Gierl (eds.). Cognitive Diagnostic Assessment for Education: Theory and Application[C]. Cambridge: Cambridge University Press. 119-145.
[58]	Zhang L. M. & L. J. Zhang. 2013. Relationships between Chinese college test takers' strategy use and EFL reading test performance: A structural equation modeling approach[J]. RELC Journal 44(1): 35-57.
[59]	范劲松、季佩英、俞理明. 2014. 语言测试效度研究的另一视角:考试的因子结构研究[J]. 外语教学理论与实践(4): 34-40.
[60]	金艳. 2012. 计算机化语言测试的效度研究——浅析计算机能力与测试构念的关系[J]. 外语电化教学(1): 11-15.
[61]	江进林、文秋芳. 2016. 基于Rasch模型的翻译测试效度研究[J]. 外语电化教学(1): 14-18.
[62]	孔菊芳. 2016. 基于过程证据的英语阅读测试效度验证[D]. 上海: 上海外国语大学.
[63]	施雅俐. 2015. 从答题过程检验CET4长篇阅读匹配题的构念效度[J]. 外语测试与教学(1): 24-31.
[64]	胥云、武尊民. 2011. 国外考试策略研究综述——五十年回顾[J]. 外语教学理论与实践(1): 43-51.
[65]	邹申、张艳莉、周越美. 2002. 阅读测试中题目类型、策略与分数的关系——TEM4考试阅读项目的答题效度研究[J]. 外语与外语教学(5): 19-22.