Genemo, Hussein (2019) A Design Science Approach to Assessment of Multi-Steps Questions in Mathematics. PhD thesis, Victoria University.

Abstract

Academic assessment that is used to assist in producing knowledge having a lasting effect should be the concern of all stakeholders. Subjective assessment contributes to achieving this type of knowledge. In particular for multi-steps questions (MSQ) human assessors play important roles. This evaluation includes assessing solution strategies, working produced from employing those strategies, and the final answer. This assessment can reveal levels of conceptual understanding displayed by students. Subjective assessment gives options to students to express their understanding of the topic. However, when computer-aided assessment (CAA) systems are used, there is much less subjectivity. These systems subdivide MSQ into stub-steps, and students can provide one step’s answer without showing their workings. This technique is similar to assessing objective questions such as multiple-choice questions (MCQ) which do not, generally, examine conceptual understanding. The inability of CAA systems to assess solutions of MSQ is considered a complex problem. To solve a complex problem, accumulating knowledge about the problem, producing solutions and acquiring ability to apply those solutions are required. Design science research (DSR), which is the paradigm to investigate complex problems- in information systems research and design science - has been used in this research. Two hundred and fifty-eight student scripts containing working with solutions of MSQ were analysed using an inductive qualitative content analysis and the quantitative survey approach. The findings from the analyses of scripts were re-analysed iteratively to produce the knowledge that contributed to the understanding of the research problem and producing solutions. This knowledge includes types of solution strategies and student errors that can also be used in designing questions to be used in CAA systems for assessing objective questions. The methodologies that were used in producing this knowledge could also be used in similar disciplines to produce similar output. A significant contribution of this research is to analyse student workings for extracting solution strategies, and this helps understanding ways of solving MSQ and obtaining these strategies. This approach does not appear to have been used elsewhere. The questionnaire that was created for measuring the significance of types of strategies and student errors is an innovative instrument. It has not been used previously and can be adapted for similar studies. The information in the analyses of student workings also provides directions when solutions of MSQ are assessed by humans. Furthermore, all the above information as well as domain knowledge was used to develop DSR constructs and models artefacts. These models represent processes and data in the student workings, which was divided into sub-tasks with each task being represented by one model. The implementation of a model in a DSR prototype artefact accomplishes the assessment of that sub-task. One of these models was implemented, overcoming difficulty in designing user interfaces that can be used without disclosing the domain knowledge students are examined. These user interfaces show an innovative way of extracting conceptual understanding of topics. Furthermore, these models are highly generalisable to a very broad class of problems.

Search Google Scholar

Repository staff login