Computer-Based, Adaptive and Behaviour-Related Assessment of Information and Communication-Related Competencies (ICT skills) in PISA

Project Description

Owing to the relevance of computers in daily life, competent handling of information and communication technologies (ICT) is a prerequisite for a successful participation in modern knowledge-based societies and thus an important goal in education. In a cooperative project, a computer-based instrument was developed aiming to test ICT skills of 15-year old students. In this context, a valid and efficient measurement of ICT skills is possible because authentic and adaptive (i.e. fitting individual competence levels) items are presented in behavior-related environments.

Objectives

1. Definition of a theoretical framework
   The first partial goal aimed at the preparation of a theoretical framework concept to define and organise the domain of ICT-related skills. Based on PISA questionnaires concerning familiarity with computers ('ICT Familiarity Questionnaire‘)  and existing theoretical concepts, new and so far neglected aspects such as cooperative types of collaboration were taken into account in the theoretical framework. Assisted by psychological theories and models, the internal structure of ICT skills was described. The framework concept defines prototypical ICT tasks, e.g. internet-based queries for information, and describes the therefore relevant ICT skills and abilities. Furthermore, aspects defining constructs and difficulties of such ICT tasks were identified with regard to task development.
2. Item development and field testing
   Based on the theoretical framework concept, the second objective concerned the development of a survey concept, whereby representative tasks were systematically allocated to the formulated skills and abilities. Task types can be categorised and ordered according to different characteristics (e.g. characteristics defining difficulty, context, mode of reaction), preparing the basis for a systematic task development. Accordingly, trained item developers constructed items containing simulations of ICT environments (e.g. a web browser), implementing them by using the CBA ItemBuilder, an authoring tool. Cognitive Laboratory Studies were subsequently conducted to put these items to a first qualitative test, leading to an optimisation and implementation in a large sample in the PISA 2015 Field Test.

Methodological Approach

Planned steps (1-5) of the project were oriented towards the conceptual developmental process that is typically pursued in international Large-Scale Assessments. In a first step, a theoretical framework was created (1), defining and organising the domain of ICT-related skills and serving as a foundation for test development (2). Data for scaling development and validation (3) of the measurement instruments were subject to a national supplementary study to the PISA 2015 Field Test and served as a basis for the development of the adaptive test (4). The developed instrument could thus be applied to test the role of ICT skills in computer-based assessments (5).

Project phases and steps were conducted in close cooperation of all project partners involved. Here, project partners at DIPF were particularly responsible for preparing the theoretical framework, deducting the assessment concept, developing and delivering the software for the item development (CBA ItemBuilder), technical delivery of the assessment instrument and formulating the validation approach

Funding

Federal Ministry of Education and Research (BMBF)

Co-operations

Prof. Dr. Holger Horz, Goethe-Universität Frankfurt am Main (Projektkoordination) ; Prof. Dr. Andreas Frey, Friedrich-Schiller-Universität Jena

Further Information

ZIB

Project Team

Project Details