Checklist:
Evaluative criteria for computer-delivered language learning systems
compiled during the Invitational Symposium on Assessing
and Advancing Technology Options in Language Learning
(AATOLL) February 1998
General description of the program
Name/version ___________________________
Intended users
age
o children
o high
school
o
college
o
learners with special interests (tourists, businessmen, etc.)
ability level
o
beginning
o
intermediate
o
advanced
Intended use
o self-instruction
o textbook/classroom supplement
o
combination of above
Content
o
content likely to be of interest to intended users
o instructional units logically sequenced
o lessons
build on material presented earlier
Program goals
o goals and
objectives of the program clearly described
o
program capable of meeting the stated goals
Scope
number of lessons or
instructional units _________________________________
equivalence in terms of
hours, semesters, years of instruction ______________
Documentation
o discussion of program’s goals, design, and contents
o tutorial on program’s operation
o
demonstration lesson included
o lesson
plans for use in a course or guidance for individual use
o
research results on effectiveness of the program available
o
information about independent reviews of the program
Customization
o
choices depending on user level
o users
can customize the operation of the program
o
variety of options to support different types of learners
Installation/operation
o
installation instructions clear
o
program easy to install/uninstall
How often does the system
crash? _________________________________
Is there a particular
action that causes regular crashes? _________________________________
Hardware requirements
type of computer _________________________________
version of operating
system
_________________________________
memory (RAM) _________________________________
disk space _________________________________
CD-ROM drive/DVD-ROM
drive
_________________________________
speakers _________________________________
microphone _________________________________
monitor _________________________________
sound card/video card _________________________________
network/browser
requirements
_________________________________
other _________________________________
Media
o
graphics, video and audio help users to concentrate on content
o text
easy to read
o audio
clear
o colors
and graphics crisp and clear
o video
runs smoothly
o
hypertext links work well
o
animation serves a pedagogical purpose
Feedback
o users
receive quick feedback to responses
o level
of feedback language appropriate for intended users
availability of feedback
(e.g., item-by-item, logical content break, end-of-unit or session,
learner-controlled)
_________________________________
type of feedback is available
o
indication as to whether response is correct or incorrect
o
invitation to try again
o hint
leading to correct response
o
explanation as to why a particular response is incorrect
o a
"give-up" option
o
directions to relevant part of program for review
o
intelligent feedback depending on learner response history
Evaluation/tests
o
frequency of the tests
o type
of scores is made available to users
o
explanation of test scores
o
cumulative log of scores to track progress
o
recommendations for remedial action given
o tests
and/or test scores can be printed out
Special features:
o speech
recognition
o system
requires initial training
What do users need to do
to train the system?
___________________________
_________________________________
How frequently does the
system recognize a range of native speech? _________
How does the system react
to nonnative speech? _________________________
___________________________
___________________________
How extensively is speech
recognition/processing utilized in each unit of the program?
___________________________
___________________________
function of speech
recognition/processing in this program
o voice
navigation
o
pronunciation instruction
o
speaking practice
Developer/distributor
name ___________________________
street address ___________________________
telephone ___________________________
fax ___________________________
e-mail ___________________________
URL ___________________________
Price
individual ___________________________
institutional ___________________________
o users likely to get their money’s worth
Checklist for listening software
Listening input
audio
o conditions against which the audio is played
authentic
o
tempo of the sound track natural
o
variety of voices and dialects
o
sound track supported by video or graphics
listening passages
o
passages authentic
o
topics of interest to intended users
o variety of topics
o users can choose among several passages on the same
topic
o variety of genres
o passages of an appropriate length for intended users
o vocabulary appropriate for the intended level
o syntax appropriate for the intended level
Listening activities
pre-listening activities
o program adapts to different levels of users’ prior
knowledge
o pre-listening activities that activate prior
knowledge
listening activities
o listening activities emphasize comprehension of the
passage
o listening activities authentic
o variety of listening activities
o users have a choice of listening activities
o activities motivate learners to keep listening
post-listening activities
o post-listening activities based on selected features
of passages
o features selected for special attention well chosen
o post-listening activities promote acquisition of
vocabulary
Listening strategy instruction
o explanation about the benefits of using listening strategies
o systematic practice in the use of listening
strategies
o users can see that certain tasks are more easily
accomplished when strategies are used
o choice of listening strategies depending on learning
style
o feedback on the effectiveness of strategy use
Listening tools
links to the written version
of passage
o the whole passage
o sentence-by-sentence
o phrase-by-phrase
o word-by-word
spoken glosses
o monolingual
o bilingual
o hint-type
written glosses
o monolingual
o bilingual
o hint-type
visual glosses
o images
o graphics
o videos
additional resources available
o online talking dictionary
o online written dictionary
o online reference grammar
o background information
o cultural notes
Listening interface
o ease of navigation
o navigation between audio, activities, glosses, and
tools simple
o navigation between screens fast
playback control
o users can control the speed of audio playback
playback can be stopped
o after each phrase
o after each sentence
timing
o enough time to complete activities
archiving
o users’ work can be saved
o users’ work can be printed out
Checklist for reading software
Reading input
How authentic is the
appearance of the passage?
o layout
o fonts
o illustrations
o colors
content of the reading
passages
o
reading passages authentic
o
topics of interest to intended users
o
variety of topics
o
topics not likely to become outdated
o
variety of genres
o
users can choose among several passages on the same
topic
o
length appropriate for intended users
o
vocabulary appropriate for intended users
o
syntax appropriate for intended users
Reading activities
pre-reading activities
o
program adapts to different levels of users’ prior
knowledge
o
pre-reading activities that activate prior knowledge
reading activities
o
reading activities emphasize comprehension of the
passage
o
reading activities authentic
o
variety of reading activities
o
users have a choice of reading activities
o
activities motivate learners to keep reading
post-reading activities
o
post-reading activities based on selected features
of the passages
o
features selected for special attention well chosen
o
post-listening activities promote acquisition of
vocabulary
Reading strategy instruction
o
explanation about the benefits of using reading strategies
o
systematic practice in the use of reading strategies
o
users can see that certain tasks are more easily
accomplished with strategy use
o
choice of reading strategies depending on learning
style
o
feedback on the effectiveness of strategy use
Reading tools
links available to the spoken
version of the passage
o
the whole passage
o
sentence-by-sentence
o
phrase-by-phrase
o
word-by-word
written glosses available
o
monolingual
o
bilingual
o
hint-type
spoken glosses available
o
monolingual
o
bilingual
o
hint-type
visual glosses available
o
images
o
graphics
o
videos
additional resources available
o
topical glossaries
o
background information
o
cultural information
o
online dictionary
o
online reference grammar
Reading interface
ease of navigation
o
navigation between text, activities, glosses, and
tools simple
o
navigation between screens fast
appearance of the text
o
texts preserve their original format
o
texts presented in a variety of fonts and typefaces
o
fonts and typefaces attractive and easy to read
o
illustrations and graphics aid in text comprehension
glosses
o
equal access to different types of glosses
o
glossed items marked unobtrusively
o
glosses cover up text or make it disappear
o
different types of glosses can be customized by
users
timing
o
enough time to complete activities
archiving
o
users’ work can be saved
o
users’ work can be printed out
Checklist for speaking software
Speaking input
general qualities of the input
o
authenticity
o
situational appropriateness
o
cultural appropriateness
o
interest to intended users
topics
o
topics of interest to intended users
o
variety of topics
o
topics logically sequenced
o
users have a choice of topics
vocabulary
o
vocabulary appropriate for the intended level
o
vocabulary organized into well-defined thematic
units
o
program focuses users’ attention on specific
vocabulary
o
vocabulary recycled throughout the program
grammar
o
grammar appropriate for the intended level
o
presentation of grammar logically sequenced
o
program focuses users’ attention on specific
structures
o
grammar recycled throughout the program
o
grammatical explanations clear
Speaking activities
o
speaking activities simulate real-world interactions
o
activities challenging and fun
o
activities progress from mechanical drills to
open-ended responses
o
activities require integration of linguistic, sociolinguistic,
and pragmatic skills
o
speaking activities contextualized
o
variety of activities
o
users have a choice of activities
What is the quality of
the speaking prompts? ___________________________
o
examples for the activities to be performed
responses allowed
o
only one
o
more than one
Speaking strategy instruction
o
program explains why the use of communication
strategies can be helpful
o
systematic practice in the use of communication
strategies
o
users can see that a task is more easily
accomplished when a particular communication strategy is used
o
choice of communication strategies depending on
learning style
o
feedback on effectiveness of strategy use
Speaking tools
online tools are available
o
reference grammar
o
monolingual dictionary
o
bilingual dictionary
o
talking dictionary
o
thesaurus
o
sociolinguistic and cultural notes
Speech interface
o
speech interface user-friendly
speech recognition (SR)
o
pedagogical goals of the program could be achieved
without SR
tasks associated with SR
o
creating a sentence with audio and/or visual cues
o
conversing with the system (conversation changes
depending on response)
Feedback from SR
o
Is the feedback from SR correlated with how well
something was said
o
feedback relies only on learners’ own assessment
o
users can record and compare their responses with
correct models
SR program provides feedback
on accuracy
o
in responses chosen from a closed set
o
in open-ended responses
archiving
o
users’ work can be recorded and saved
Checklist for writing software
Input for writing activities
input provided in order to
contextualize writing tasks
o
written texts
o
listening passages
o
videos
o
input at the appropriate level of difficulty
o
variety of input
o
users have a choice of type of input
Writing activities
o
writing tasks have counterparts in the real world
o
tasks at the appropriate level of difficulty
o
tasks culturally appropriate
o
writing prompts clearly and concisely worded
writing prompts specify
o
an audience
o
a function
o
a motivation for writing
o
choice of writing tasks
Writing strategy instruction
o
explanation of he benefits of using writing
strategies
o
program offers systematic practice in the use of
writing strategies
o
users can see that certain writing tasks are more
easily accomplished when strategies are used
o
choice of writing strategies depending on learning
style
o
feedback on the effectiveness of strategy use
Writing tools
macro-aides available
o
online templates for brainstorming writing tasks
o
online templates for organizing the writing tasks
o
online samples for imitation
micro-aides available
o
online bilingual dictionary
o
online thesaurus
o
online grammar reference
editing aides available
o
online spell checker
o
online grammar checker
Writing interface
o
enough time given for task completion
o
program can save successive drafts of users’ work
o
users’ work can be printed out
o
program can keep track of user queries
o
program includes a network component for group work
or peer editing
Checklist for vocabulary learning software
Input for vocabulary learning
basis for the choice of
vocabulary included in the program
o
frequency
o
association with a particular topic or topics
o
association with a particular textbook
vocabulary appropriate for the
intended users in terms of
o
level of difficulty
o
interest
o
usefulness
Vocabulary learning activities
o
activities challenging and fun
o
users have choice of difficulty level
o
activities relate new vocabulary to vocabulary
previously introduced
o
users have a choice of activities
vocabulary learning activities
offered
o
repetition of the word
o
recognition/matching activities pairing the word
with its native language equivalent
o
pairing the word with its target language synonym
o
pairing the word with its target language antonym
o
finding the related word in a list
o
pairing the word with a picture
o
pairing a word with a video
o
listening to the word being pronounced and finding
it in a list
production activities
o
giving the target language equivalent for a native
language word
o
finding the right word to describe a picture
o
finding a target language synonym for a word
o
finding a target language antonym for a word
o
putting a word in the appropriate semantic group
o
producing derivationally related words (e.g., noun
from a verb)
o
using the word in a sentence
o
games
Vocabulary learning strategy instruction
o
explanation of different kinds of mnemonic
strategies
o
explanation of why the use of certain strategies can
improve retention of vocabulary
o
systematic practice in the use of mnemonic
strategies
o
users can see that a task is more easily
accomplished when a mnemonic strategy is used
o
users offered a choice of mnemonic strategies
depending on their learning style
o
feedback on whether the use of a particular mnemonic
strategy was effective or not
Vocabulary learning tools
o
vocabulary search engine
glosses provided
o
monolingual
o
bilingual
o
hint-type
o
examples of usage
o
visual
o
audio
o
video
Vocabulary learning interface
vocabulary search engine
glosses provided
o
monolingual
o
bilingual
o
hint-type
o
examples of usage
o
visual
o
audio
o
video
Checklist for software to teach pronunciation
Pronunciation input
words presented
o
in citation form only
o
in different contexts
o
suprasegmental features included in the program
o
intonation included in the program
o
native-speaker models include a variety of voices
and dialects
Pronunciation activities
o
activities well-sequenced
o
activities challenging and fun
o
activities progress from easier to more complex
o
users have control over selection of activities
o
opportunities to review the material
tasks associated with the use
of speech recognition
o
reading one word at a time
o
filling in a blank in a sentence
o
reading from a selection of several words
o
reading one sentence at a time
o
reading from a selection of 2-5 sentences
o
reading from a selection of 6 or more sentences
o
creating a sentence with audio and/or visual cues
Pronunciation interface
speech recognition system
reaction to nonnative speech
o
recognizes despite mistakes
o
asks for repetition
o
does not recognize
o
recognition system robust to various disfluencies
o
recognition system sensitive to ambient noise
feedback recognition system
provides
o
correlates with how well something was said
o
relies only on learners’ own assessment
o
users can record and compare their responses with
correct models
mistakes in learner responses
are marked and displayed
o
accuracy of vowels and consonants
o
accuracy of speech rhythm and timing
o
accuracy of intonation contours
method by which visual
pronunciation feedback presented
o
speaker’s face
o
vocal tract
o
spectrum information
o
speech waveforms
Checklist for software to teach script
Script input
basis for selection of
characters
o
frequency
o
topic association
o
textbook association
o
information about the history of the writing system
o
general description of the writing system and its
distinguishing features
Script learning activities
characters (or symbols)
practiced
o
in isolation
o
in combinations
o
in meaningful sequences
o
from the more simple to the more complex
o
characters (or symbols) presented earlier are
recycled throughout the program
o
variety of activities
o
users have control over choice of activities
Script strategy instruction
o
program explains what techniques are helpful in
learning to write the target language script
o
program provides systematic practice in the use of
these techniques
Script learning tools
o
characters (or symbols) linked to their
pronunciation
links to glosses
o
monolingual
o
bilingual
o
images
o
video
Script learning interface
o
program shows stroke order of the characters or
symbols
o
animation of how characters or symbols are written
o
search engine
©1998 Second Language Teaching & Curriculum Center, University of Hawai`i