Situation: is the diagram part of an open learning package or is it for use in a traditional classroom situation?
Effectiveness test: How well does the diagram work in a specific instructional context.
Evaluate during the development process and make revisions, contact:
- content expert,
- artist,
- media consultant (book editor, video director),
- instructional designer,
- target learners.
In case of conflicts: try out the diagrams under video condition to convince your partners.
The content expert has to see the final version - in any case!
Be aware: the graphic design (artist) may not coincide with the instructional design.
Check artwork carefully and be tactful when proposing alterations.
Show your early diagram drafts to your colleagues.
Try each diagram out in the medium for which it is destined.
Draw up a checklist for the particular medium.
Learner-based evaluation
Evaluation by the potential users is a critical step: trade-offs in terms of time and other resources have to be made.
Employ this three-stage process:
- Detailed one-to-one evalutation with individual learners (not in the context of use of the materials).
- Small group evaluation: a limited sample representing the main types of target learners (including context).
- A field trial in which the material is evaluated in the actual context of use.
A finer grained approach for evaluating diagrams:
- diagram alone,
- diagram in conjunction with its accompanying instructional material,
- diagram in the context of its use.
Questions:
- Do learners consider the diagram to be helpful?
- Do learners process the diagram sufficiently deeply overall?
- Do learners understand and remember the material presented?
- Can learners apply the material presented?
- Are there particular aspects of the diagram that learners find distracting, annoying, confusing?
- Do learners explore the information provided in the diagram in a productive fashion?
- Do learners deal effectively with the various transformations that have been used in the diagram?
- Can learners identify all the entities shown in the diagram and
assign them in the diagram's explanation fo the subject matter?
- Can learners locate all the important relations that exist among the diagram's entities?
- Do learners integrate material in the diagram effectively with its accompanying materials?
- Can learners translate the relations which the diagram depicts in visuo-spatial terms
into their intended real-life meanings?
- Can learners go beyond the diagram and use it to deal successfully with the actual subject matter of the diagram?
Both the general and specific questions fit into the following categories of evaluation:
1 Reactions: the opinions learners have of a diagram.
2 Processing: the way learners work with a diagram.
3 Direct instructional results: how learners grasp the subject matter.
4 Performance: the new capacities learners can demonstrate.
Learners' opinions
Safeguards when collecting informations how learners react.
Provide a relaxed atmosphere for one-to-one or small group evaluations
to encourage the learners to speak freely about the diagrams.
Distance yourself from ownership, otherwise they may hold back.
Provide learners with a framework that stimulates critical thinking about key aspects:
- pencil and paper instruments: rating scales, questionnaires, open-ended responses),
- interviews using probes which target specific aspects of learner's response,
- observations of the way learners react when using the diagram to perform some task, make them think aloud.
Learners' opinion can help provide answers to these questions:
- How approachable did learners find the diagram?
- Did learners find it difficult to deal with?
- How interesting and aesthetically pleasing is the diagram?
- Did it have a professional feel about it?
- Was the amount of information given in the diagram about right?
- Do learners consider this or any diagram is necessary?
- How effective is the diagram in addressing its stated objective?
- How well does the diagram fulfil the learners' needs at the place where it occurred in the instruction?
- How well is the diagram linked to other parts of the instructional resource?
Learners' results
What do the learners know after dealing with the diagram?
Be careful about the nature of these tests:
- Adding correct labels to unlabelled diagram.
- Matching a list of labels to the correct parts of the diagram.
- Generating labels by the learners.
- Generating the diagram to match given labels?
Learners' processing
The evaluation may investigate:
- External behaviour: what people do with an instructional diagram.
- Internal processing: why they behave as they do.
Collecting information about the results of a diagram processing is easier
than collecting it about the processing itself.
Because diagrams are used as an alternative to words, it can be difficult for a person
to say much about the internal processing.
Thinking aloud while working thru a diagram may deliver useful informations. Record it with a video camera.
Ask the learner to point to parts of the diagram thinking about.
After own reviews, let the learner watch the recorded session and comment.
Suitable probes for collecting verbal informations:
- Give me a summary in your own words of what this diagram is about.
- Show me the main sections of this diagram.
- Tell me what this part of the diagram shows.
Hints for collecting non-verbal informations:
- Ask the learner to copy and then recall the diagram: examine the nature of the changes occuring upon copying or realling.
- Which entities were lost or changed in the processing? Which preserved?
- What sort of changes did entities undergo during processing: shape, size, orientation, position, rendering?
- At what levels are breakdowns in structure of the diagram evident: overall structure, local details, mid-level structure.
Learners' performance
Evaluate the effectiveness of a diagram by having the person carry out that procedure using the diagram as a guide.
Record the difficulties: note on a copy of the diagram the parts causing difficulties. Retrace the steps of the learner.
Probes may be used to check how the learner recognizes, locates and manipulates:
- Recognition: Present the object dismantled: Please, show which is part x.
- Location: Present the object assembled: Please, show where to find part x.
- Manipulation: Present the object partly dismantled: Please, remove the part x.
The learner should be able to make explicit the connections between the diagram and the other form of representation.
Gathering information
Entities
How successful do learners deal with entities?
- Isolating individual entities: knowing where each entity starts and ends, what constitutes an entity.
- Identifying each entity: knowing what that entity stands for in the diagram.
- Characterizing the properties of the entities, interpreting their significance within the diagram.
Activities:
- Marking the boundaries of each of the labelled or listed pieces with a different colour.
- Indentifying larger composite entities, as well as the individual items it comprises.
- Several passes for marking: first basic level, then what goes together.
- Explicit naming may be or may not be important.
- Asking questions that require the learner to go beyond what is explicitly depicted.
Relations
The explanatory power of a diagram results from the presentation of the relations between the entities.
How effectively can learners deal with these relations? Techniques:
- Asking learners to point to the more important parts of the diagram,
the less important parts, the parts that are in some way different from the rest of the diagram. Ask why.
- Having learners explain what is wrong with a version of the diagram that has been deliberately altered.
- Giving learners an incomplete version of the diagram: what is missing, what the missing parts belong to?
- Asking learners to extend a diagram beyond its current scope in time or space.
- Having learners classify parts of the diagram into categories that they invent themselves:
what seems to belong together in some way?
- Asking learners to justify their groupings.
Field testing diagrams
The diagram designer likes to know how the target learners treat the diagram in their instructional context.
Since extrapolating from the decontextualized testing (for diagram development)
to realities of instructional application in context may be dangerous.
Only field trial can show some or all influences of realtime application.
Chris Mueller (prolingua@access.ch)
