Didier Strasser

Université de Genève
Faculté de Psychologie et des Sciences de l'Education
Summary

Büchel (e.g. Schlatter, Büchel, & Thomas, 1997) and collaborators have developed the Analogical Reasoning Learning Test (ARLT). The principle underlying this dynamic assessment approach is to provide standardized error-specific hints depending on the student's responses. The application of the hinting procedures places a considerable memory and attention load on the examiner. There is not only the need to manage the test session and to record the scores, but the examiner also has to keep track of the test instructions to correctly apply the hints that match each response. These difficulties led to the development of a computer-assisted test protocol. This tool includes electronic access to the scoring sheets, the manual of instructions and the error-specific hinting procedures.

Abstract

In 1995, Büchel (e.g. Schlatter, Büchel, & Thomas, 1997) and collaborators have developed the Analogical Reasoning Learning Test (ARLT) that is specifically designed to evaluate the analogical reasoning of the students with moderate mental retardation. It is consistent a dynamic assessment paradigm (Budoff & Friedman, 1964), because the test procedure doesn't value or quantify the student's previously acquired knowledge or skills (static test paradigm), but helps to determine his or her learning potential. The goal of the ARLT is to discriminate between students with mental retardation who profit from the provided hints and increase their performance on analogical tasks from those who do not progress with the hints. The examiner therefore provides hierarchical and standardized hints, according to the student's responses.

A validity study of the ARLT items showed that the management of the entire test session (e.g. lay out and remove test pictures), the simultaneous reading of the instructions to ensure error-specific hints and the related scoring generated a certain stress for the examiners. The correct application of the heavy mental load is mainly felt by novice examiners and it affects the ease and spontaneity with which the hints are provided. Moreover, there is a risk of mistakes in scoring the student's performance.

The examiner's mental load is greatly reduced with the development of a unique computer-assisted support tool that includes the scoring sheets, the manual and the hints. It was important to keep some constraints in mind that are related to the specific use of this protocol. First of all, since the test is mostly used in institutions, it was necessary to use a portable computer. This restricts the dimension of the windows on the computer screen. Secondly, the examiner is not always an expert computer user, which means that the management of files needs to be transparent and self-regulated for the user. Finally, the goal of the computer-assisted protocol is to decrease the attention overload of the examiner. It is therefore essential to avoid that this new tool leads to new, unexpected side effects that tap on attentional resources. Presenting the content of three sheets (notation sheet, manual and hints) on a single screen could have negative effects. In fact, studies about the influence of computer screen design on user performance showed that the density of information on a screen has a direct consequence on the rate of searching time and errors (Schneiderman, 1992; NASA, 1980). Moreover, Tullis (1988, p. 387) recommends that "the designer should ensure that each screen contains only the information that is actually needed by the users to perform the expected task." He supports this recommendation by the fact that "the temptation to put additional data on the screen just because it is available should be avoided, since extra clutter clearly degrades the users' ability to extract the relevant information" (ibid.). For the ARLT protocol it seems essential that the given information should not be permanently on the screen and that the instructions, the illustrations and the hints should stay hidden until they are needed. Such needs may be defined intentionally (prompted by the user) or managed by the software.

The Computer-Assisted Protocol (Strasser, 1996a; 1996b; 1996c) that will be presented in the poster session is a software utility based on the Macintosh version of Excel by Microsoft. However, it is not necessary to know the functioning of this program and its particularities in order to correctly use the protocol because the potential of Excel is used in a transparent way for the user: there are reduced menu bars and self-regulated basic routines, such as saving files with a new name, saving during the work, and so on.

The protocol takes three specific levels of use into account. The first one is the experimentation level that offers a directed and restricted environment. At this level, the user can't access the sheets on which the results are kept to prevent an accidental modification that could affect the scientific reliability. Moreover, the vertical and horizontal scrollbars, the panels and the toolbars are removed in order to prevent the user from navigating freely between the windows and to make sure that his actions are guided by the student's results.

The second level concerns the results. The authorized user can access the scoring and result sheets as well as the possibility to print them. The configuration being the same as in the experimentation level, the navigation between the sheets is possible with specific menu items.

Finally, there is the design level which allows restricted access to the standard environment of Microsoft Excel, the Visual Basic for Excel modules and the dialog boxes.

The computer-assisted protocol has not yet been evaluated empirically. However, first informal tests showed some limitations due to slow processing on computers with processor slower than 75 MHz (Strasser, 1996a). The reason for this problem has been detected and it is sure that the choice of Excel as a programming tool was not judicious. To enable the use of the protocol on computers with more modest processors, it would be necessary to rebuild it with a program that is specialized in the creation of softwares, such as Authorware from Macromedia.

Besides these technical questions, the computer-assisted protocol seems to satisfy the goal it was made for, even if an empirical evaluation still needs to be done before it is distributed. It is important to clarify if using a computer in a test session does not provoke mental overload for a lambda user.