Modeling an Adaptive Test Based on the Results of Classical Testing
Abstract
Testing is widely used by teachers and lecturers of universities, both for current and for attestation control of knowledge. Compared to traditional assessment methods, testing is the fastest, least time-consuming and more objective method. However, testing is often controversial. Some call it the "lottery", recalling cases when a successful “C” student completed a test with "excellent", but an "A" student could not cope with the same test. Testing is a random process in which we can only guarantee the probabilities, not its outcome. To reduce the impact of random factors, the test should comply with the recommendations of the item response theory (IRT). Test parameters such as: the method of testing, the content and number of tasks in the test, their distribution according to the difficulty of performing, the rating scale, etc., should be set based on the goals of testing, taking into account the preparedness of the audience. Usually, in pedagogical practice, classical testing algorithms are used, but their goal: to assess the level of preparedness of the subjects with maximum accuracy, does not correspond to the purpose of testing: assign each subject to one of the given categories. More suitable for this purpose is adaptive testing based on the Bayesian algorithm, the purpose of which is to most accurately estimate the probabilities of belonging to each category. This publication is devoted to a comparative analysis of classical and adaptive testing methods. To compare them, an adaptive test was simulated on the results of classical testing. To compare the reliability of the test results, scatter matrices were calculated. The experiment showed that the reliability of adaptive testing is higher than that of classical testing, despite the fact that the number of completed tasks is 60% of the tasks in classical testing. The results of the work show the importance of introducing adaptive methods into computer testing systems used in knowledge control.
References
2. Belous N.V., Kutsevich I.V. Model' adaptivnogo kontrolya znanij [Model of adaptive control of knowledge]. Radioelektronika, informatika, upravlenie = Radio Electronics. Computer Science. Control. 2010;(1):39-44. (In Russ., abstract in Eng.) EDN: SAZQLJ
3. Kornilov D.E. Sravnitel'nyj analiz adaptivnoj i klassicheskoj modeli testirovaniya [Comparative analysis of adaptive and classical testing models]. Perspektivy razvitiya informacionnyh tekhnologij = Prospects for the development of information technologies. 2014;(19):119-124. (In Russ., abstract in Eng.) EDN: SEFOEJ
4. Oppl S., Reisinger F., Eckmaier A. et al. A flexible online platform for computerized adaptive testing. International Journal of Educational Technology in Higher Education. 2017;14:2. https://doi.org/10.1186/s41239-017-0039-0
5. Nurakhmetov D. Reinforcement Learning Applied to Adaptive Classification Testing. In: Veldkamp B., Sluijter C. (eds.) Theoretical and Practical Advances in Computer-based Educational Measurement. Methodology of Educational Measurement and Assessment. Cham: Springer; 2019. p. 325-366. https://doi.org/10.1007/978-3-030-18480-3_17
6. [ 5]Gamage S.H.P.W., Ayres J.R., Behrend M.B. et al. Optimising Moodle quizzes for online assessments. International Journal of STEM Education. 2019;6:27. https://doi.org/10.1186/s40594-019-0181-4
7. Romanenko D. Computer Testing as a Form of Students Knowledge Control. CEUR Workshop Proceedings. 2020;2770:196-203. Available at: https://ceur-ws.org/Vol-2770/paper23.pdf (accessed 11.02.2023).
8. Maslova O., Kuklina A. Adaptive testing model based on IRT techniques. The Scientific Heritage. 2020;(43):41-44. EDN: AGLIOQ
9. Riza L.S., Nurjanah N.S., Wihardi Y. The development of computer adaptive test and item response theory with 4 parameters based the logistics model. Journal of Physics: Conference Series. 2019;1280:032033. https://doi.org/10.1088/1742-6596/1280/3/032033
10. Wulandari F., Hadi S., Haryanto. Computer-based Adaptive Test Development Using Fuzzy Item Response Theory to Estimate Student Ability. Computer Science and Information Technology. 2020;8(3):66-73. https://doi.org/10.13189/csit.2020.080302
11. Hu Z.F., Lin L., Wang Y.H., Li J.W. The Integration of Classical Testing Theory and Item Response Theory. Psychology. 2021;12:1397-1409. https://doi.org/10.4236/psych.2021.129088
12. Kimura T. The impacts of computer adaptive testing from a variety of perspectives. Journal of Educational Evaluation for Health Professions. 2017;14:12. https://doi.org/10.3352/jeehp.2017.14.12
13. Sijtsma K. Developments in Measurement of Persons and Items by Means of Item Response Models. Behaviormetrika. 2001;28:65-94. https://doi.org/10.2333/bhmk.28.65
14. Bortolotti S.L.V., Tezza R., de Andrade D.F. et al. Relevance and advantages of using the item response theory. Quality & Quantity. 2013;47:2341-2360. https://doi.org/10.1007/s11135-012-9684-5
15. Koliada M., Bugayova T., Miklashevich N. Intelligent Testing Systems Based on Adaptive Algorithms. CEUR Workshop Proceedings. 2020;2770:172-186. Available at: https://ceur-ws.org/Vol-2770/paper21.pdf (accessed 11.02.2023).
16. Almansour F.M., Alroobaea R., Ghiduk A.S. An Empirical Comparison of the Efficiency and Effectiveness of Genetic Algorithms and Adaptive Random Techniques in Data-Flow Testing. IEEE Access. 2020;8:12884-12896. https://doi.org/10.1109/ACCESS.2020.2966433
17. Starichenko B.E., Gizatullin M.G., Istomina E.A. Assessment of the level of readiness and quality of test materials using the online form test_results. Pedagogical Education in Russia. 2016;(7):104-112. (In Russ., abstract in Eng.) https://doi.org/10.26170/po16-07-15
18. Buravlev A.I., Pereverzev V.Yu., Golyaeva N.V. Ocenka kachestva podgotovki na osnove bajesovskoj procedury adaptivnogo kriterial'no-orientirovannogo testirovaniya [Evaluation of the quality of training based on the Bayesian procedure of adaptive criterion-oriented testing]. Srednee professional'noe obrazovanie = Secondary vocational education. 2006;(9):18-21. (In Russ., abstract in Eng.) EDN: KTOGIR
19. Gusyatnikov V.N., Sokolova T.N., Bezrukov A.I., Kayukova I.V., Adaptive model for testing several competencies based on the Bayes algorithm. Modern high technologies. 2022;(1):40-46. (In Russ., abstract in Eng.) https://doi.org/10.17513/snt.39007
20. Nekhaev I., Zhuykov I., Manukyants S., Maslennikov A. Applying Bayesian Network to Assess the Levels of Skills Mastering in Adaptive Dynamic OER-Systems. In: Silhavy R., Silhavy P., Prokopova Z. (eds.) Software Engineering Perspectives in Intelligent Systems. CoMeSySo 2020. Advances in Intelligent Systems and Computing. Vol. 1294. Cham: Springer; 2020. p. 1090-1116. https://doi.org/10.1007/978-3-030-63322-6_94
21. Bezrukov A.I., Grakholskaya L.V. Imitacionnaya model' dlya vybora strategii adaptivnogo testirovaniya [Simulation model for choosing an adaptive testing strategy]. Matematicheskoe i komp'yuternoe modelirovanie v ekonomike, strahovanii i upravlenii riskami = Mathematical and computer modeling in economics, insurance and risk management. 2020;(5):145-151. (In Russ., abstract in Eng.) EDN: KYXXJM
22. Bloem R., Fey G., Greif F. et al. Synthesizing adaptive test strategies from temporal logic specifications. Formal Methods in System Design. 2019;55:103-135. https://doi.org/10.1007/s10703-019-00338-9
23. Rodríguez I., Rubio D., Rubio F. Complexity of adaptive testing in scenarios defined extensionally. Frontiers of Computer Science. 2023;17:173206. https://doi.org/10.1007/s11704-022-1673-9
24. Kostikov A., Vlasenko K., Lovianova I., Volkov S., Kovalova D., Zhuravlov M. Assessment of Test Items Quality and Adaptive Testing on the Rasch Model. In: Ermolayev V., et al. (eds.) Information and Communication Technologies in Education, Research, and Industrial Applications. ICTERI 2021. Communications in Computer and Information Science. Vol. 1698. Cham: Springer; 2022. p. 252-271. https://doi.org/10.1007/978-3-031-20834-8_12
25. Tagirova L.F., Zubkova T.M. Intelligent adaptive testing system. Scientific and Technical Journal of Information Technologies, Mechanics and Optics. 2023;23(4):757-766. (In Russ., abstract in Eng.) https://doi.org/10.17586/2226-1494-2023-23-4-757-766
This work is licensed under a Creative Commons Attribution 4.0 International License.
Publication policy of the journal is based on traditional ethical principles of the Russian scientific periodicals and is built in terms of ethical norms of editors and publishers work stated in Code of Conduct and Best Practice Guidelines for Journal Editors and Code of Conduct for Journal Publishers, developed by the Committee on Publication Ethics (COPE). In the course of publishing editorial board of the journal is led by international rules for copyright protection, statutory regulations of the Russian Federation as well as international standards of publishing.
Authors publishing articles in this journal agree to the following: They retain copyright and grant the journal right of first publication of the work, which is automatically licensed under the Creative Commons Attribution License (CC BY license). Users can use, reuse and build upon the material published in this journal provided that such uses are fully attributed.
