Development and Psychometrical Validation of a Computer-Assisted Language Learning Evaluation Scale: An Ecolinguistic Approach

Document Type : Original Article

Authors

Department of Applied Linguistics and TEFL, Payame Noor University, Tehran, Iran.

Abstract

From the complex dynamic systems theory perspective, evaluating CALL pedagogy involves miscellaneous variables. It seems the ecosystem of the CALL environment necessitates an array of extralinguistic (Ecolinguistically situated) issues to be considered in any appraisal of the CALL milieu. To this end, this paper aims to develop and validate a scale for CALL evaluation called the Ecolinguistics CALL Evaluation Scale (ECES). It enframes the complex processes of language use in CALL and technology mediation as facets of a complex adaptive system (CAS). The psychometric testing of a questionnaire was undertaken with 219 EFL academics to investigate the sociocultural, sociopolitical, and ideological values and norms within the CALL ecosystem, thereby representing the tripartite interaction of human beings, Ecolinguistics, and CALL. The model being probed included 14 components of ECES. Confirmatory Factor Analysis (CFA) was run using LISREL Software (Ver. 8.80) to examine the trait structure of the ECES questionnaire. Results yielded that all items had significant contributions to their constructs, and their respective t-values all indicated that the contribution of items to their constructs was also statistically significant. This scale might equip a more streamlined method that is readily adaptable to a variety of contexts, stakeholders, and criteria.

Keywords

References

Bachman, L. F. (2005). Statistical analysis for language assessment. (2nd ed.). Cambridge University Press. https://doi.org/10.1017/CBO9780511667350
Bandura, A., & National Inst of Mental Health. (1986). Social foundations of thought and action: A social cognitive theory. Prentice Hall. https://doi.org/10.4135/9781446221129.n6
Bax, S. (2003). CALL – Past, present, and future. System, 31, 13–28. https://doi.org/10.1016/S0346-251X(02)00071-4
Beatty, K. (2010). Teaching and researching: Computer-Assisted Language Learning (2nd ed.). Pearson Education Limited. https://doi.org/10.4324/9781315833774
Beatty, P. C., & Willis, G. B. (2007). Research synthesis: The practice of cognitive interviewing. Public opinion quarterly, 71(2), 287-311. https://doi.org/10.1093/poq/nfm006
Boateng, G. O., Neilands, T. B., Frongillo, E. A., Melgar-Quiñonez, H. R., & Young, S. L. (2018). Best practices for developing and validating scales for health, social, and behavioral research: a primer. Frontiers in Public Health, 6, 149. https://doi.org/10.3389/fpubh.2018.00149
Burston, J. (2003). Software selection: A primer on sources and evaluation. CALICO Journal, 21(1), 29-40. https://doi.org/10.1558/cj.v21i1.29-40
Chau, J., & Lee, A. (2014). Technology-enhanced language learning (TeLL): An update and a principled framework for English for Academic Purposes (EAP) courses.Canadian Journal of Learning and Technology 40(1). http://dx.doi.org/10.1558/cj.v21i1.29-40
Chapelle, C. (2001). Computer applications in second language acquisition: Foundations for teaching, testing and research. Cambridge University Press. https://doi.org/10.1017/CBO9781139524681
Chapelle, C. (2007). Challenges in evaluation of innovation: Observations from technology research. Innovation in Language Learning and Teaching, 1(1), 30-45. http://dx.doi.org/10.2167/illt041.0
Chapelle, C. (2010). The spread of computer-assisted language learning. Language Teaching, 43(1), 66-74. https://doi.org/10.1017/S0261444809005850
Chen, S. (2016). Language and ecology: A content analysis of Ecolinguistics as an emerging research field. Ampersand, 3(1), 108-116. https://doi.org/10.1016/j.amper.2016.06.002
Clark, L. A., & Watson, D. (1995). Constructing validity: basic issues in objective scale development. Psychological Assessment, 7(3), 309–319. https://psycnet.apa.org/doi/10.1037/14805-012
Creswell, J. W., & Plano Clark, V. L. (2018). Designing and Conducting Mixed Methods Research (3rd ed.). SAGE. ISBN:9781483347011, 148334701X https://doi.org/10.1558/cj.v30i0.272-279
Colpaert, J. (2013). Peripatetic considerations on research challenges in CALL. CALICO Journal, 30, 272-279.
Dafouz, E., & Smit, U. (2020). ROAD-MAPPING English medium education in the internationalized university. Palgrave Macmillan. http://dx.doi.org/10.14198/raei.2021.34.10
Derni, A. (2008). The ecolinguistic paradigm: An integrationist trend in language study. The International Journal of Language Society and Culture, 24, 21-30.
DeVellis, R. F. (2017). Scale Development: Theory and Applications (4th ed.). SAGE Publications, Inc. https://doi.org/10.1177/014662169101500413
Salkind, N. J. (2007). Questionnaires. In Encyclopedia of measurement and statistics (Vol. 0, pp. 809-812). Sage Publications, Inc., https://doi.org/10.4135/9781412952644
 Felix, U. (2008). The unreasonable effectiveness of CALL: What have we learned in two decades of research? ReCALL, 20(2), 141–161. https://doi.org/10.1017/S0958344008000323
Garrett, N. (2009). Computer-assisted language learning trends and issues revisited: Integrating innovation. The Modern Language Journal, 93 (focus issue), 719-740. https://doi.org/10.1111/j.1540-4781.2009.00969.x
Hartshorn, K. J., McMurry, B. L., West, R. E., Rich, P., Williams, D. D., & Anderson, (2016). An Evaluation Framework for CALL. Retrieved from https:// dx.doi.org/10.24093/awej/call10.8
Hilsenroth, M. J., Blagys, M. D., Ackerman, S. J., Bonge, D. R., & Blais, M.A(2005). Measuring psychodynamic-interpersonal and cognitive-behavioural techniques: development of the comparative psychotherapy process scale. Psychotherapy: Theory, Research, Practice, Training, 42(3), 340–356. http://dx.doi.org/10.10p. 37/0033-3204.42.3.340
Hollan, J., Hutchins, E., Kirsh, D., 2000. Distributed cognition: toward a new foundation for human–computer interaction. ACM Transactions on Human-Computer Interaction 7 (2), 174–196. https://doi.org/10.1145/353485.353487
Hubbard, P. (2008). Twenty-five years of theory in the CALICO Journal. Calico Journal, 25(3), 387-399. https://doi.org/10.1558/cj.v25i3.387-399
Hubbard, P., & Colpaert, J. (2019). Toward transdisciplinarity in computer-assisted language learning. calico journal, 36(2), 81-99. https://doi.org/10.1558/cj.37499
Ivankova, N. V., Creswell, J. W., & Stick, S. L. (2006). Using mixed-methods sequential explanatory design: From theory to practice. Field methods, 18(1), 3-20. https://doi.org/10.1177/1525822X05282260
Kessler, G. (2018). Technology and the future of language teaching. Foreign language annals, 51(1), 205-218. https://doi: 10.1111/flan.12318
Kline, R. B. (2016). Principles and practice of structural equation modelling. 4th ed. Guilford Press. https://doi.org/10.4236/health.2013.511254
Larsen-Freeman, D. (1997). Chaos/complexity science and second language acquisition. Applied Linguistics, 18(2), 141–165. https://doi: 10.1093/applin/18.2.141
Larsen-Hall, J. (2016). A guide to doing statistics in second Language learning research Using SPSS. (2nd ed.). Routledge. https://doi.org/10.4324/9781315775661
Levy, M. (1997). Computer-assisted language learning: Context and conceptualization. Oxford University Press. https://doi.org/10.1017/S0958344000003803.ISBN:9780198236313,019823631X
Levy, M., & Stockwell, G. (2013). CALL dimensions: Options and issues in computer-assisted language learning. Routledge. https://doi.org/10.4324/9780203708200
Lee, J. S., & Drajati, N. A. (2020). Willingness to communicate in digital and non-digital EFL contexts: Scale development and psychometric testing. Computer Assisted Language Learning, 33(7), 688-707. https://doi.org/10.1080/09588221.2019.1588330
Liu, G. Z., Liu, Z. H., & Hwang, G. J. (2011). Developing multidimensional evaluation criteria for English learning websites with university students and professors. Computers & Education, 56(1), 65-79. https://doi.org/10.1016/j.compedu.2010.08.019
Lobo, R., Petrich, M., & Burns, S. K. (2014). Supporting health promotion practitioners to undertake evaluation for program development. BMC Public Health, 14, 1-8. http://www.biomedcentral.com/1471-2458/15/1315
Marek, M.W., &Wu,W.-C. V. (2014). Environmental factors affecting computer-assisted language learning success: A complex dynamic systems conceptual model. Computer Assisted Language Learning, 27(6), 1–19. http://dx.doi.org/10.1080/09588221.2013.776969
Mackey, W. F. (2001). The ecology of language shift. In A. Fill & P. Mühlhäusler (Eds.), The ecolinguistics reader: Language, ecology, and environment (pp. 67-74). Continuum. ISBN:9780826481733, 0826481736
MacKenzie, S. B., Podsakoff, P. M., & Podsakoff, N. P. (2011). Construct measurement and validation procedures in MIS and behavioral research: integrating new and existing techniques. MIS Quarterly, 35(2), 293–334. https://doi.org/10.2307/23044045
McKeown-Ice, R., & Dendinger, R. (2000). Socio-political-cultural foundations of environmental education. The Journal of Environmental Education, 31(4), 37-45. https://doi.org/10.1080/00958960009598650
McMurry, B. L., Williams, D. D., Rich, P. J., & Hartshorn, K. J. (2016). An Evaluation Framework for CALL. TESL-EJ, 20(2), n2.
Moshagen M (2021). semPower: Power Analyses for SEM_. R package version 1.2.0, https://doi.org/10.1080/10705511.2014.950896
Novawan, A., Walker, S. A., & Ikeda, O. (2024). The new face of technology-enhanced language learning (TELL) with artificial intelligence (AI): Teacher perspectives, practices, and challenges. Journal of English in Academic and Professional Communication, 10(1), 1-18. https://doi.org/10.25047/jeapco.v10i1.4565
Nunnally, J. C. (1975). Psychometric theory. 25 Years Ago, and Now. Educational Researcher, 4(10), 7-21 https://doi.org/10.3102/0013189X004010007
Petticrew, M., & Roberts, H. (2008). Systematic reviews in the social sciences: A practical guide. John Wiley & Sons. https://doi:10.1002/9780470754887
Piaget, J. (1971). The theory of stages in cognitive development. McKay. ISBN: 9780679301639,0679301631
Smith, G. T. (2005). On construct validity: issues of method and measurement. Psychological assessment, 17(4), 396. https://doi: 10.1037/1040-3590.17.4.396
Schulze, M. (2017). Complexity approaches to computer-assisted language learning. In S. L. Thorne & P. A. Schneider (Eds.), Encyclopedia of language and education (pp. 1-12). Springer.
Soleimani, H. (2020). Computer-assisted language learning: Theory and practice. Payame Noor University Press.
Steffensen, S. V., & Fill, A. (2014). Ecolinguistics: the state of the art and future horizons. Language sciences, 41, 6-25. http://dx.doi.org/10.1016/j.langsci.2013.08.003
Susser, B. (2001). A defense of checklists for courseware evaluation. ReCALL, 13(2), 261-276. https://doi.org/10.1017/S0958344001001021a
Tomlinson, B. (Ed.). (2003). Developing materials for language teaching. A&C Black. ISBN:9780826459176, 082645917X
Vandewaetere, M., & Desmet, P. (2009). Introducing psychometrical validation of questionnaires in CALL research: The case of measuring attitude towards CALL. Computer Assisted Language Learning, 22(4), 349-380. https://doi.org/10.1080/09588220903186547
Villada, E. G. (2009). CALL evaluation for early foreign language learning: A review of the literature and a framework for evaluation. CALICO Journal, 26(2), 363-389. https://doi:10.1558/cj.v26i2.363-389
Vygotsky, L. S. (1978). Mind in society: The development of higher psychological Processes. Harvard University Press. ISBN: 9780674576292
Wang, Y., & Stockwell, G. (2023). Social Justice and Technology in Second Language     Education. Iranian Journal of Language Teaching Research, 11(3 (Special Issue)), 1-18. doi: 10.30466/ijltr.2023.121403
Warschauer, M., & Kern, R. (2000). Network-based language teaching concepts and practice. Cambridge University Press. https://doi.org/10.1017/CBO9781139524735
Warschauer, M. (2002). Reconceptualizing the digital divide. First Monday. https://doi.org/10.5210/fm.v7i7.967
Warschauer, M. (2004). Technology and social inclusion: Rethinking the digital divide. MIT Press. http://dx.doi.org/10.7551/mitpress/6699.001.0001
Warschauer, M. (2004). Technological change and the future of CALL. In S. Fotos & C. Brown (Eds.), New perspectives on CALL for second and foreign language classrooms (pp. 15–25). Lawrence Erlbaum Associates. https://doi.org/10.4324/9781410610775
Warschauer, M. (2011). Learning in the cloud. Teachers College Press. ISBN: 9780807752494
Zhu, X., Raquel, M., & Aryadoust, V. (2019). Structural equation modeling to predict performance in English proficiency tests. In Quantitative Data Analysis for Language Assessment Volume II (pp. 101-126). Routledge.ISBN:9781315187808
Iranian Journal of Language Teaching Research
Volume 13, Issue 1
March 2025
Pages 123-146

Files

History

  • Receive Date: 27 October 2024
  • Accept Date: 22 February 2025

Share

How to cite

Statistics