文章论述了PEG、IEA、IntelliMetric、e-rater、BETSY这五种具有代表性的自动作文评阅(AES)系统的原理、特征、功能、优缺点,分析比较其共性和差异,论证其发展过程,总结和展望未来发展的若干特点:设计有助于提高学习者认知能力和辩证性思维能力的AES系统;评判重点从语言和结构转向论点思辨和修辞效果;能够对各种文体类型的作文进行评定;开发对写作过程提供形成性评估的AES系统;利用机器学习技术,设计开放式AES系统平台;开发和利用可进行人机对话的反馈模式;交叉学科的合作在系统发展中的作用将更为突出;逐步建立对多种语言的评分反馈功能。
This paper gives an overview of the principles, features and functions of the most well known automated essay scoring systems, including PEG, IEA, e rater and Criterion, IntelliMetric and MY Access!, and BETSY. Reliabilities of these systems are analyzed, and strengths and weaknesses of each of the systems are compared and contrasted. The paper analyzes the future development of AES systems on the basis of the discussion of former researches and the model of cognitively based assessment of writing competency. It summarize the orientation of development in seven aspects: design of AES systems on cognitively based assessment model of writing competence; shift of emphasis from surface features of grammar and structure to underlying features of critical thinking and rhetorical effect; expansion of subject areas, with focus on both English language arts and scientific reasoning; development of a new genre of AES software that can provide meaningful and effective formative feedback to assist writing process; use of machine learning technology to develop an open AES system that can address new problems automatically; enhancement of feedback effect via machine student dialogue; integration and cooperation of various disciplines of studies in the field of AES.
Agerri, R., X. Artola, Z. Beloki, G. Rigau & A. Soroa. 2015. Big data for natural language processing: A streaming approach [J]. Knowledge-Based Systems 79:36-42.
Attali, Y. 2015. Reliability-based feature weighting for automated essay scoring [J]. Applied Psychological Measurement 39(4):303-313.
Burstein, J. 2003. The E-rater® scoring engine: Automated essay scoring with natural language processing [A]. In M. D. Shermis & J. C. Burstein (eds.). Automated Essay Scoring: A Cross-Disciplinary Perspective [C]. Mahwah: Lawrence Erlbaum Associates. 113-121.
Burstein, J., M. Chodorow & C. Leacock. 2004. Automated essay evaluation: The criterion online writing service [J]. AI Magazine 25:27-35.
Burstein, J. & D. Marcu. 2000. Benefits of modularity in an automated essay scoring system [A]. Proceedings of the COLING-2000 Workshop on Using Toolsets and Architectures to Build NLP Systems [C]. Luxembourg: Association for Computational Linguistics.44-50.
Burstein J., D. Marcu & K. Knight. 2003. Finding the WRITE stuff: Automatic identification of discourse structure in student essays [J]. IEEE Intelligent Systems 18:32-39.
Chapelle, C.A., E. Cotos & J. Lee. 2015. Validity arguments for diagnostic assessment using automated writing evaluation [J]. Language Testing 32(3):385-405.
Cheville, J. 2004. Automated scoring technologies and the rising influence of error [J]. English Journal 93:47-52.
Chung, K.W.K. & H.F. O'Neil. 1997. Methodological approaches to online scoring of essays [OL]. [2016-07-06]. http://www.cse.ucla.edu/products/reports/tech461.pdf.
Deane, P., T. Quinlan & I. Kostin. 2011. Automated scoring within a developmental, cognitive model of writing proficiency [R]. Princeton: Educational Testing Service.
Dikli, S. 2006. An overview of automated scoring of essays [J]. Journal of Technology, Learning, and Assessment (5):1-35.
Elliot, S. 2002. A study of expert scoring, standard human scoring and IntelliMetric scoring accuracy for statewide eighth grade writing responses [R]. Newtown: Vantage Learning.
Elliot, S. 2003. IntelliMetric: From here to validity [A]. In M.D. Shermis & J. Burstein (eds.). Automated Essay Scoring: A Cross-Disciplinary Perspective [C]. Mahwah: Lawrence Erlbaum.71-86.
Enright, M. & M. Quinlan. 2010. Complementing human judgment of essays written by English language learners with e-rater scoring [J]. Language Testing 27:317-334.
Foltz, P. W. 1996. Latent Semantic Analysis for text-based research [J]. Behavior Research Methods, Instruments and Computers 28:197-202.
Kaplan, R.M., S.E. Wolff, J. Burstein, C. Lu, D.A. Rock & B.A. Kaplan. 1998. Scoring essays automatically using surface features [R]. Princeton: Educational Testing Service.
Kukich, K. 2000. Beyond automated essay scoring [J]. IEEE Intelligent Systems 15:22-27.
Landauer, T., K. & S. Dumais. 1997. A solution to Plato's problem: The latent semantic analysis theory of the acquisition, induction, and representation of knowledge [J]. Psychological Review 104:211-140.
Landauer, T.K., P.W. Foltz & D. Laham. 1998. Introduction to latent semantic analysis [J]. Discourse Processes 25:259-284.
Landauer, T.K., D. Laham & P.W. Foltz. 2000. The intelligent essay assessor [J]. IEEE Intelligent Systems 15:27-31.
Landauer, T. K., P. W. Foltz & D. Laham. 2004. What is LSA? [OL]. [2016-07-06]. http://lsa.colorado.edu/whatis.html.
Landauer, T. K., D. Laham & P. W. Foltz. 2003. Automatic essay assessment [J]. Assessment in Education 10(3):295-308.
Lemaire, B. & P. Dessus. 2001. A system to assess the semantic content of student essays [J]. Educational Computing Research 24:305-306.
Lottridge, S.M., E.M. Schulz & H.C. Mitzel. 2013. Using automated scoring to monitor reader performance and detect reader drift in essay scoring [A]. In M.D. Shermis & J. Burstein (eds.). Handbook of Automated Essay Evaluation: Current Applications and New Directions [C]. New York: Routledge. 233-250.
Mayfield, E. & C.P. Rose. 2013. LightSIDE: Open source machine learning for text [A]. In M.D. Shermis & J. Burstein (eds.). Handbook of Automated Essay Evaluation: Current Applications and New Directions [C].New York: Routledge. 124-135.
Page, E.B. 1966. The imminence of grading essays by computer [J]. Phi Delta Kappan 47:238-243.
Page, E. & N. S. Peterson. 1995. The computer moves into essay grading: Updating the ancient test [J]. Phi Delta Kappan 76:561-565.
Quinlan, T., D. Higgins & S. Wolff. 2009. Evaluating the construct-coverage of e-rater Scoring Engine [R]. Princeton: ETS.
Rich, C.S., M.C. Schneider & J.M D'Brot. 2013. Applications of automated essay evaluation in west Virginia [A]. In M.D. Shermis & J. Burstein (eds.). Handbook of Automated Essay Evaluation: Current Applications and New Directions [C]. New York: Routledge. 99-123.
Rudner, L.M. & T. Liang. 2002. Automated essay scoring using Bayes' theorem [J]. The Journal of Technology, Learning, and Assessment 1(2):3-21.
Rudner, L., V. Garcia & C. Welch. 2005. An evaluation of IntellimetricTM essay scoring system using responses to GMAT AWA prompts [R]. McLean: Graduate Management Admission Council.
Shermis, M.D. & J. Burstein. 2003. Introduction [A]. In M.D. Shermis & J. Burstein (eds.). Automated Essay Scoring: A Cross-disciplinary Perspective[C]. Mahwah: Lawrence Erlbaum. xiii-xvi.
Sukkarieh, J. Z. & E. Bolge. 2010. Building a textual entailment suite for the evaluation of automatic content scoring technologies [OL]. [2016-07-06]. http://www.lrec-conf.org/proceedings/lrec2010/pdf/310_Paper.pdf
Valencia-García, R. & F. García-Sánchez. 2013. Natural language processing and human-computer interaction [J]. Computer Standards & Interfaces 35:415-416.
Valenti, S., F. Neri. & A. Cucchiarelli. 2003. An overview of current research on automated essay grading [J]. Journal of Information Technology Education 2:319-330.
Vantage Learning. 2005. How IntelliMetricTM Works [OL]. [2016-07-06]. http://www.cengagesites.com/academic/assets/sites/4994/WE_2_IM_How_IntelliMetric_Works.pdf.
Wang, Y., J. Zhang & Y. Xu. 2014. Research on construction of natural language processing system based on semantic web ontology [J]. Journal of Chemical and Pharmaceutical Research 6(12): 291-296.
Weigle, S. C. 2013. English language learners and automated scoring of essays: Critical considerations [J]. Assessing Writing 18:85-99.
Warschauer, M. 2014. Next-generation automated feedback in support of iterative writing and scientific argumentation [R]. Irvine: University of California.
Warschauer, M. & D. Grimes. 2008. Automated essay scoring in the classroom [J]. Pedagogies 3(1): 22-36.
Warschauer, M. & P. Ware. 2006. Automated writing evaluation: Defining the classroom research agenda [J]. Language Teaching Research 10(2):1-24.
黄志娥、谢佳莉、荀恩东. 2014. HSK自动作文评分的特征选取研究[J]. 计算机工程与应用(6):118-126.
刘明杨、秦兵、刘挺. 2016. 基于文采特征的高考作文自动评分智能[J]. 计算机与应用(1):1-8.
梁茂成、文秋芳. 2007. 国外作文自动评分系统评述及启示[J]. 外语电化教学(5):18-24.
殷杰、董佳蓉. 2008. 论自然语言处理的发展趋势[J]. 自然辩证法研究(3):31-37.
