Checklist:
Evaluative criteria for computer-delivered language learning systems
compiled during the Invitational Symposium on Assessing
and Advancing Technology Options in Language Learning
(AATOLL) February 1998
General description of the program
Name/version ___________________________
Intended users
age
o children
o high
school
o
college
o
learners with special interests (tourists, businessmen, etc.)
ability level
o
beginning
o
intermediate
o
advanced
Intended use
o self-instruction
o textbook/classroom supplement
o
combination of above
Content
o
content likely to be of interest to intended users
o instructional units logically sequenced
o lessons
build on material presented earlier
Program goals
o goals and
objectives of the program clearly described
o
program capable of meeting the stated goals
Scope
number of lessons or
instructional units _________________________________
equivalence in terms of
hours, semesters, years of instruction ______________
Documentation
o discussion of program’s goals, design, and contents
o tutorial on program’s operation
o
demonstration lesson included
o lesson
plans for use in a course or guidance for individual use
o
research results on effectiveness of the program available
o
information about independent reviews of the program
Customization
o
choices depending on user level
o users
can customize the operation of the program
o
variety of options to support different types of learners
Installation/operation
o
installation instructions clear
o
program easy to install/uninstall
How often does the system
crash? _________________________________
Is there a particular
action that causes regular crashes? _________________________________
Hardware requirements
type of computer _________________________________
version of operating
system
_________________________________
memory (RAM) _________________________________
disk space _________________________________
CD-ROM drive/DVD-ROM
drive
_________________________________
speakers _________________________________
microphone _________________________________
monitor _________________________________
sound card/video card _________________________________
network/browser
requirements
_________________________________
other _________________________________
Media
o
graphics, video and audio help users to concentrate on content
o text
easy to read
o audio
clear
o colors
and graphics crisp and clear
o video
runs smoothly
o
hypertext links work well
o
animation serves a pedagogical purpose
Feedback
o users
receive quick feedback to responses
o level
of feedback language appropriate for intended users
availability of feedback
(e.g., item-by-item, logical content break, end-of-unit or session,
learner-controlled)
_________________________________
type of feedback is available
o
indication as to whether response is correct or incorrect
o
invitation to try again
o hint
leading to correct response
o
explanation as to why a particular response is incorrect
o a
"give-up" option
o
directions to relevant part of program for review
o
intelligent feedback depending on learner response history
Evaluation/tests
o
frequency of the tests
o type
of scores is made available to users
o
explanation of test scores
o
cumulative log of scores to track progress
o
recommendations for remedial action given
o tests
and/or test scores can be printed out
Special features:
o speech
recognition
o system
requires initial training
What do users need to do
to train the system?
___________________________
_________________________________
How frequently does the
system recognize a range of native speech? _________
How does the system react
to nonnative speech? _________________________
___________________________
___________________________
How extensively is speech
recognition/processing utilized in each unit of the program?
___________________________
___________________________
function of speech
recognition/processing in this program
o voice
navigation
o
pronunciation instruction
o
speaking practice
Developer/distributor
name ___________________________
street address ___________________________
telephone ___________________________
fax ___________________________
e-mail ___________________________
URL ___________________________
Price
individual ___________________________
institutional ___________________________
o users likely to get their money’s worth
Checklist for listening software
Listening input
audio
o conditions against which the audio is played
authentic
o
tempo of the sound track natural
o
variety of voices and dialects
o
sound track supported by video or graphics
listening passages
o
passages authentic
o
topics of interest to intended users
o variety of topics
o users can choose among several passages on the same
topic
o variety of genres
o passages of an appropriate length for intended users
o vocabulary appropriate for the intended level
o syntax appropriate for the intended level
Listening activities
pre-listening activities
o program adapts to different levels of users’ prior
knowledge
o pre-listening activities that activate prior
knowledge
listening activities
o listening activities emphasize comprehension of the
passage
o listening activities authentic
o variety of listening activities
o users have a choice of listening activities
o activities motivate learners to keep listening
post-listening activities
o post-listening activities based on selected features
of passages
o features selected for special attention well chosen
o post-listening activities promote acquisition of
vocabulary
Listening strategy instruction
o explanation about the benefits of using listening strategies
o systematic practice in the use of listening
strategies
o users can see that certain tasks are more easily
accomplished when strategies are used
o choice of listening strategies depending on learning
style
o feedback on the effectiveness of strategy use
Listening tools
links to the written version
of passage
o the whole passage
o sentence-by-sentence
o phrase-by-phrase
o word-by-word
spoken glosses
o monolingual
o bilingual
o hint-type
written glosses
o monolingual
o bilingual
o hint-type
visual glosses
o images
o graphics
o videos
additional resources available
o online talking dictionary
o online written dictionary
o online reference grammar
o background information
o cultural notes
Listening interface
o ease of navigation
o navigation between audio, activities, glosses, and
tools simple
o navigation between screens fast
playback control
o users can control the speed of audio playback
playback can be stopped
o after each phrase
o after each sentence
timing
o enough time to complete activities
archiving
o users’ work can be saved
o users’ work can be printed out
Checklist for reading software
Reading input
How authentic is the
appearance of the passage?
o layout
o fonts
o illustrations
o colors
content of the reading
passages
o
reading passages authentic
o
topics of interest to intended users
o
variety of topics
o
topics not likely to become outdated
o
variety of genres
o
users can choose among several passages on the same
topic
o
length appropriate for intended users
o
vocabulary appropriate for intended users
o
syntax appropriate for intended users
Reading activities
pre-reading activities
o
program adapts to different levels of users’ prior
knowledge
o
pre-reading activities that activate prior knowledge
reading activities
o
reading activities emphasize comprehension of the
passage
o
reading activities authentic
o
variety of reading activities
o
users have a choice of reading activities
o
activities motivate learners to keep reading
post-reading activities
o
post-reading activities based on selected features
of the passages
o
features selected for special attention well chosen
o
post-listening activities promote acquisition of
vocabulary
Reading strategy instruction
o
explanation about the benefits of using reading strategies
o
systematic practice in the use of reading strategies
o
users can see that certain tasks are more easily
accomplished with strategy use
o
choice of reading strategies depending on learning
style
o
feedback on the effectiveness of strategy use
Reading tools
links available to the spoken
version of the passage
o
the whole passage
o
sentence-by-sentence
o
phrase-by-phrase
o
word-by-word
written glosses available
o
monolingual
o
bilingual
o
hint-type
spoken glosses available
o
monolingual
o
bilingual
o
hint-type
visual glosses available
o
images
o
graphics
o
videos
additional resources available
o
topical glossaries
o
background information
o
cultural information
o
online dictionary
o
online reference grammar
Reading interface
ease of navigation
o
navigation between text, activities, glosses, and
tools simple
o
navigation between screens fast
appearance of the text
o
texts preserve their original format
o
texts presented in a variety of fonts and typefaces
o
fonts and typefaces attractive and easy to read
o
illustrations and graphics aid in text comprehension
glosses
o
equal access to different types of glosses
o
glossed items marked unobtrusively
o
glosses cover up text or make it disappear
o
different types of glosses can be customized by
users
timing
o
enough time to complete activities
archiving
o
users’ work can be saved
o
users’ work can be printed out
Checklist for speaking software
Speaking input
general qualities of the input
o
authenticity
o
situational appropriateness
o
cultural appropriateness
o
interest to intended users
topics
o
topics of interest to intended users
o
variety of topics
o
topics logically sequenced
o
users have a choice of topics
vocabulary
o
vocabulary appropriate for the intended level
o
vocabulary organized into well-defined thematic
units
o
program focuses users’ attention on specific
vocabulary
o
vocabulary recycled throughout the program
grammar
o
grammar appropriate for the intended level
o
presentation of grammar logically sequenced
o
program focuses users’ attention on specific
structures
o
grammar recycled throughout the program
o
grammatical explanations clear
Speaking activities
o
speaking activities simulate real-world interactions
o
activities challenging and fun
o
activities progress from mechanical drills to
open-ended responses
o
activities require integration of linguistic, sociolinguistic,
and pragmatic skills
o
speaking activities contextualized
o
variety of activities
o
users have a choice of activities
What is the quality of
the speaking prompts? ___________________________
o
examples for the activities to be performed
responses allowed
o
only one
o
more than one
Speaking strategy instruction
o
program explains why the use of communication
strategies can be helpful
o
systematic practice in the use of communication
strategies
o
users can see that a task is more easily
accomplished when a particular communication strategy is used
o
choice of communication strategies depending on
learning style
o
feedback on effectiveness of strategy use
Speaking tools
online tools are available
o
reference grammar
o
monolingual dictionary
o
bilingual dictionary
o
talking dictionary
o
thesaurus
o
sociolinguistic and cultural notes
Speech interface
o
speech interface user-friendly
speech recognition (SR)
o
pedagogical goals of the program could be achieved
without SR
tasks associated with SR
o
creating a sentence with audio and/or visual cues
o
conversing with the system (conversation changes
depending on response)
Feedback from SR
o
Is the feedback from SR correlated with how well
something was said
o
feedback relies only on learners’ own assessment
o
users can record and compare their responses with
correct models
SR program provides feedback
on accuracy
o
in responses chosen from a closed set
o
in open-ended responses
archiving
o
users’ work can be recorded and saved
Checklist for writing software
Input for writing activities
input provided in order to
contextualize writing tasks
o
written texts
o
listening passages
o
videos
o
input at the appropriate level of difficulty
o
variety of input
o
users have a choice of type of input
Writing activities
o
writing tasks have counterparts in the real world
o
tasks at the appropriate level of difficulty
o
tasks culturally appropriate
o
writing prompts clearly and concisely worded
writing prompts specify
o
an audience
o
a function
o
a motivation for writing
o
choice of writing tasks
Writing strategy instruction