آمار
| تعداد نشریات | 43 |
| تعداد شمارهها | 1,226 |
| تعداد مقالات | 15,041 |
| تعداد مشاهده مقاله | 50,585,688 |
| تعداد دریافت فایل اصل مقاله | 13,674,981 |
Constructing and Validating a Q-Matrix for Cognitive Diagnostic Analysis of a Reading Comprehension Test Battery | ||
| Journal of English Language Teaching and Learning | ||
| مقاله 2، دوره 10، شماره 21، شهریور 2018، صفحه 1-31 اصل مقاله (826.72 K) | ||
| نوع مقاله: Research Paper | ||
| نویسندگان | ||
| Mohammad Alavi* ؛ Fatemeh Ranjbaran | ||
| University of Tehran | ||
| چکیده | ||
| Of paramount importance in the study of cognitive diagnostic assessment (CDA) is the absence of tests developed for small-scale diagnostic purposes. Currently, much of the research carried out has been mainly on large-scale tests, e.g., TOEFL, MELAB, IELTS, etc. Even so, formative language assessment with a focus on informing instruction and engaging in identification of student’s strengths and weaknesses to guide instruction has not been conducted in the Iranian English language learning context. In an attempt to respond to the call for developing diagnostic tests, this study explored developing a cognitive diagnostic reading comprehension test for CDA purposes. To achieve this, initially, a list of reading attributes was prepared based on the literature and then the attributes were used to construct 20 reading comprehension items. Then seven content raters were asked to identify the attributes of each item of the test. To obtain quantitative data for Q-matrix construction, the test battery was administered to 1986 students of a General English Language Course at the University of Tehran, Iran. In addition, 13 students were recruited to participate in think-aloud verbal protocols. On the basis of the overall agreement of the content raters’ judgements concerning the choices of attributes and results of think-aloud verbal protocol analysis, a Q-matrix that specified the relationships between test items and target attributes was developed. Finally, to examine the CDA of the test, the Fusion Model, a type of cognitive diagnostic model (CDM), was used for diagnosing the participants' strengths and weaknesses. Results suggest that nine major reading attributes are involved in these reading comprehension test items. The results obtained from such cognitive diagnostic analyses could be beneficial for both teachers and curriculum developers to prepare instructional materials that target specific weaknesses and inform them of the more problematic areas to focus on in class in order to plan for better instruction. | ||
| کلیدواژهها | ||
| Attributes؛ Cognitive Diagnostic Assessment؛ Cognitive Diagnostic Model؛ Fusion model؛ Think-aloud verbal protocol؛ Q-Matrix | ||
| مراجع | ||
|
Alderson, J. C. (2000). Assessing reading. Cambridge: Cambridge University Press. Alderson, J. C. (2010). “Cognitive diagnosis and Q-Matrices in language assessment”: A Commentary, 96-103. Bachman, L. F., & Palmer, A. S. (1996). Language testing in practice. Oxford: London. Birch, B. M. (2002). English L2 reading: Getting to the bottom. Routledge. Birenbaum, M., Kelly, A. E., & Tatsuoka, K. K. (1993). Diagnosing knowledge states in algebra using the rule-space model. Journal of Research in Mathematics Education, 24, 442-459. Buck, G., & Tatsuoka, K. (1998). Application of the rule-space procedure to language testing: examining attributes of a free response listening test. Language Testing, 15(2), 119–157. Cohen, A. D., & Upton, T. A. (2006). Strategies in responding to the new TOEFL reading tasks (TOEFL Monograph No. MS-33). Princeton, NJ: ETS. De la Torre, J. (2009). A cognitive diagnosis model for cognitively based multiple-choice options. Applied Psychological Measurement, 33(3), 163-183. DiBello, L. V., Stout, W. F., & Roussos, L. (1995). Unified cognitive psychometric assessment likelihood-based classification techniques. In P. D. Nichols, S. F. Chipman, & R. L. Brennan (Eds.), Cognitively diagnostic assessment (pp. 361–390). Hillsdale, NJ: Erlbaum. DiBello, L., & Stout, W. (2008). Arpeggio documentation and analyst manual. Chicago: Applied informative assessment research enterprises (AIARE)—LLC. DiBello, L., & Stout, W. (2008). Arpeggio suite, version 3.1. 001 [Computer program]. Chicago: Applied informative assessment research enterprises (AIARE)—LLC. Embretson, S.E., Gorin, J. (2001). Improving construct validity with cognitive psychology principles. Journal of Educational Measurement, 38(4), 343-368. Fletcher, J. M. (2006). Measuring reading comprehension. Scientific Studies of Reading, 10(3), 323-330. Francis, D.J., Snow, C.E., August, D., Carlson, C.D., Miller, J., & Iglesias, A. (2006). Measures of reading comprehension: A latent variable analysis of the diagnostic assessment of reading comprehension, Scientific Studies of Reading, 10(3), 301-322. Goodman, D. P., & Hambleton, R. K. (2004). Student test score reports and interpretive guides: Review of current practices and suggestions for future research. Applied Measurement in Education, 7(2), 145-220. Hartz, S. M. (2002). A Bayesian framework for the unified model for assessing cognitive abilities: Blending theory with practicality. Dissertation Abstracts International: Section B: The Sciences and Engineering, 63(2-B), 864. Hartman, H. J. (2001). Developing students’ metacognitive knowledge and skills. In H. J. Hartman (Ed.). Metacognition in learning and instruction: Theory, Research and Practice (pp. 33-68). Dordrecht, the Netherlands: Kluwer. Huang, T.W., & Wu, P.C. (2013). Classroom-based cognitive diagnostic model for a teacher-made fraction-decimal Test. Educational Technology & Society, 16 (3), 347–361. Jang, E. E. (2005). A validity narrative: Effects of reading skills diagnosis on teaching and learning in the context of NG TOEFL. Available from ProQuest Dissertations and Theses database. (AAT 3182288) Jang, E. E. (2009). Demystifying a Q-matrix for making diagnostic inferences about L2 reading skills. Language Assessment Quarterly, 6(3), 210-238. Kim, A. Y. (2011). Examining second language reading components in relation to reading test performance for diagnostic purposes: A Fusion model approach (Doctoral dissertation, Teachers College, Columbia University). Kim, A. Y. (2015). Exploring ways to provide diagnostic feedback with an ESL placement test: Cognitive diagnostic assessment of L2 reading ability. Language Testing. [0265532214558457]. Lee, Y. W., & Sawaki, Y. (2009). Application of three cognitive diagnosis models to ESL reading and listening assessments. Language Assessment Quarterly, 6(3), 239-263. Leighton, J. P., Gierl, M. J., & Hunka, S. M. (2004). The attribute hierarchy method for cognitive assessment: A variation on Tatsuoka's Rule‐Space approach. Journal of Educational Measurement, 41(3), 205-237. Leighton, J. P., & Gierl, M. J. (2007). Cognitive diagnostic assessment for education: Theory and applications. Cambridge University Press. Li, H. (2011). Evaluating language group differences in the subskills of reading using a cognitive diagnostic modeling and differential skill functioning approach [Doctoral dissertation, The Pennsylvania State University]. Li, H., & Suen, H. K. (2013). Constructing and Validating a Q-Matrix for Cognitive Diagnostic Analyses of a Reading Test. Educational Assessment, 18(1), 1-25. Lumley, T. (1993). The notion of sub-skills in reading comprehension test: An EAP example. Language Testing, 10(3), 211-234. Mislevy, R. J. (1996). Test theory reconceived. Journal of Educational Measurement, 33(4), 379-416. Mislevy, R. J. (1994). Evidence and inference in educational assessment. Psychometrika, 59, 439–483. Mislevy, R. J., Steinberg, L. S., & Almond, R. G. (2002). Design and analysis in task-based language assessment. Language Testing. Special Issue: Interpretations, Intended Uses, and Designs in Task-based Language, 19(4), 477–496. Mislevy, R.J. (2006). Cognitive psychology and educational assessment. Educational measurement, 4, 257-305. Patz, R. J., & Junker, B. W. (1999). Applications and extensions of MCMC in IRT: Multiple item types, missing data, and rated responses. Journal of Educational and Behavioral Statistics, 24, 342–366. Pellegrino, J. C., & Chudowsky, N. N. & Glaser, R. (2001). Knowing what students know: The science and design of educational assessment. (National research council’s committee on the foundation of assessment) National Academy Press; Washington D.C. Roussos, L. A., DiBello, L. V., Stout, W. F., Hartz, S. M., Henson, R. A., & Templin, J. H. (2007). The fusion model skills diagnostic system. In J. Leighton & M. Gierl (Eds.), Cognitive diagnostic assessment for education: Theory and applications (pp. 275–318). New York, NY: Cambridge University Press. Rupp, A. A., Ferne, T., & Choi, H. (2006). How assessing reading comprehension with multiple-choice questions shapes the construct: A cognitive processing perspective. Language Testing, 23(4), 441-474. Rupp, A. A., Templin, J., & Henson, R. A. (2012). Diagnostic measurement: Theory, methods, and applications. Guilford Press. Sawaki, Y., Kim, H. J., & Gentile, C. (2009). Q-matrix construction: Defining the link between constructs and test items in large-scale reading and listening comprehension assessments. Language Assessment Quarterly, 6(3), 190-209. Sheehan, K., & Mislevy, R. (1990). Integrating cognitive and psychometric models to measure document literacy. Journal of Educational Measurement, 27, 255-272. Snow, R. E., & Lohman, D. F. (1989). Implications of cognitive psychology for educational measurement. American Council on Education. Stiggins, R., Arter, J., & Chappuis, S. (2004). Classroom assessment for student learning: Doing it right—using it well. Dover, NH: Assessment Training Institute. Svetina, D., Gorin, J.S. & Tatsuoka, K.K. (2011) Defining and comparing the reading comprehension construct: A cognitive-psychometric modelling approach, International Journal of Testing, 11:1, 1-23. Tatsuoka, K. K. (1995). Architecture of knowledge structure and cognitive diagnosis: A statistical pattern recognition and classification approach. In P. D. Nichols, S. F. Chipman, and R. L. Brennan (Eds.), Cognitively diagnostic assessment (pp. 327-361). Hillsdale, NJ: Lawrence Erlbaum Associates. Templin, J. L. (2004). Generalized linear mixed proficiency models for cognitive diagnosis. [Available from ProQuest Dissertations and Theses database. (AAT 3160960)] Urquhart, S., & Weir, C. J. (1998). Reading in a second language: Process, product and practice. New York: Longman. | ||
|
آمار تعداد مشاهده مقاله: 16 تعداد دریافت فایل اصل مقاله: 12 |
||