Developing useful evaluation practices in college foreign language programs, 2007 NFLRC Summer Institute
* Following are short statements and descriptions of projects, all of which were produced by the participants in the NFLRC Summer institute. Note that participants will be adding to this information over the next year (2007~2008) as their projects progress. If you would like to print this page, please download the original booklet.[download pdf file]
• Introduction to the Summer Institute
• Joint statement on the value of evaluation
• Strategies for culture change
1. Developing and improving study abroad programs
- Russian: University of Washington
- Multiple languages: Linfield College
2. Effectiveness of innovation
- Korean: Yale University
- Spanish: University of Missouri St. Louis
3. Understanding program value
- German: Georgetown University, Hunter
College CUNY
4. Institutional program review
- Multiple languages: Mount Saint Mary's College
5. Learner needs and curriculum improvement
- Arabic summer program: Georgetown University
- Spanish: U of South Carolina Beaufort
6. Program alignment with external benchmarks
- French: University of Oregon
- Multiple languages: Central Michigan University
7. Program development and advocacy
- Spanish: Cal. State University Monterey Bay
- Multiple languages: Cal. State University Los Angeles
8. Assessing student learning outcomes
- Spanish: University of Iowa
- Multiple languages: Duke University, University
of Evansville,
University of Florida, University
of Maryland Baltimore
9. Teacher and GTA development
- Arabic: University of Arizona
- Italian: Johns Hopkins University
Introduction
From May 28 through June 6, 2007, 24 faculty members from a diversity of
college foreign language departments across the U.S. participated in
the 2007 Summer Institute of the National Foreign Language Resource Center
at the University of Hawai‘i. The focus of this institute was Developing
useful evaluation practices in college foreign language programs. Sponsored
by the NFLRC (http://www.nflrc.hawaii.edu), in collaboration with John
Norris’s federally funded Foreign Language Program Evaluation Project
(http://www.nflrc.hawaii.edu/evaluation), the goal of the institute was
to help college foreign language administrators and faculty engage in
useful, practical, and effective program evaluations to meet a variety
of purposes. A secondary goal was to elevate the discourse about assessment
and evaluation in higher education, and to engender nationwide improvements
in the contributions made by foreign language education. In an era of
zealous accountability testing, and in light of inevitable changes in
the educational and societal roles being played by the foreign languages,
the hope was to develop and support a cadre of FL professionals to lead
the way towards a rationale and useful evaluation practice.
Participants in the institute surpassed all expectations. Working through
the Memorial Day holiday and into the weekend, they received training
in program evaluation concepts, models, and methods; they translated
and applied
evaluation ideas to the very real challenges of contemporary college
FL education contexts; and they developed extended plans for the use
of evaluation
in response to unique questions in their own particular language programs.
Above and beyond these planned institute activities, participants also
engaged in extended discussions about the value of language education
in the U.S., they articulated fundamental tensions that define much
of the
FL teaching and learning landscape, and they forged potential solutions
based on sound evaluative practices. They even adopted a name for themselves:
the Faculty Working Group on Foreign Language Program Evaluation.
In this booklet, the Faculty Working Group on Foreign Language Program Evaluation summarize three immediate outcomes of their work during the institute. First, they provide a statement on the Value of Evaluative Thinking and Action in Foreign Language Programs. Second, they offer broad Strategies for Culture Change in Program Evaluation. Third, they present one-page overviews of their program-specific evaluation work, as one way of demonstrating the range of possible contributions to be made by evaluation in college FL education. It is their hope that these collected ideas will serve as a springboard for dialogue and action within the FL disciplines.
On behalf of all of the summer institute’s organizers and facilitators, we congratulate the participants on their hard work and excellent achievements, and we look forward to sustained collaborations with them in the service of useful foreign language program evaluation.
John Norris, Director
Yukiko Watanabe, Assistant Director
NFLRC Summer Institute 2007
The Value of Evaluative Thinking and Action in Foreign Language Programs
A joint statement by the Faculty Working Group on Foreign Language Program Evaluation
This statement is the result of discussions among the participants in the NFLRC Summer Institute 2007. The intent is to articulate the value of evaluative thinking and action to foreign language education.
Evaluative thinking and action provides a framework for discussion in programs
or departments about fundamental questions of program effectiveness. These
discussions can have a democratizing and unifying effect—democratizing
because all voices are heard, and unifying because the process leads to
communication and consensus building. Collaborative discussion and action
that involves all stakeholders results in a heightened commitment of all
participants to the vitality of the program, thus contributing to a sense
of academic community.
The evaluation process allows faculty members to understand the program
as a whole and to articulate to themselves and others what they want
students to achieve in the areas of knowledge, skills, and dispositions.
By identifying
strengths and weaknesses, they formulate a plan (or plans) of action
to increase program effectiveness and maximize student learning. The
goal
is to make the learning process more efficient and to create a well-articulated
curriculum that is responsive to changing circumstances, all within a
cyclical process of innovation and evaluation.
Evaluative thinking and action has further benefits. It enables departments
to address in action-oriented ways common problems at the program level,
such as low enrollments in some languages, attrition at various levels,
and difficulties in the curricular transition from lower-division to
upper-division courses. It offers opportunities for individual faculty
members to engage
in professional development activities, such as scholarship in teaching
and learning and improving teaching practices through ongoing reflection.
It can increase communication across departments, leading to cross-pollination
between disciplines and opportunities for collaboration with colleagues
on evaluation projects, as well as professional activities in other
areas.
Beyond the department level, evaluative thinking and action enables
faculty members to enhance the profile of their program or department
within
the institution by establishing themselves as leaders in evaluation
initiatives and showcasing the accomplishments of their evaluation-related
projects.
Such leadership activities position the program or department well
in requests
for support (e.g., funding, faculty lines). Finally, the ability
to demonstrate cycles of innovation and evaluation empowers foreign
language
professionals,
enabling them to make a strong case for the unique contributions
of language studies in a pluralist and globalized world.
Strategies for Culture Change in Program Evaluation
A joint statement by the Faculty Working Group on Foreign Language Program Evaluation
This statement is the result of discussions among participants in the NFLRC Summer Institute 2007. The intent is to encourage the foreign language field to recognize program evaluation as indispensable for enhancing student learning and program quality, and to enable the field to articulate and demonstrate—internally and externally—the unique contributions of language studies in a pluralist and globalized world.
Strategies for changing perceptions of evaluation and enhancing value of evaluation
• Focus on program improvement as a goal of program evaluation.
•
Emphasize the usefulness of evaluation for: (1) student learning, (2) program
articulation, (3) departmental collaboration, and (4) academic community.
•
Highlight the public, participatory, and inclusive nature of the evaluation
process.
•
Link evaluation goals to stated institutional priorities.
Strategies for encouraging faculty-led evaluation
• Build on program information (curriculum, syllabi, final exams,
papers, etc.) and systematize evaluation work already conducted in the
department.
•
Lead institutional evaluation efforts by example; forge alliances across
the institution; draw on available institutional resources.
•
Appropriately recognize and incentivize evaluation work within the department
and the institution.
•
Integrate evaluation into standard administrative, curricular, and teaching
practices.
•
Pursue professional development opportunities and external funding.
•
Generate and showcase successful examples of evaluation.
Strategies for professional organizations to enhance useful evaluation
• Recognize and disseminate successful models of program evaluation.
•
Develop policy statements on useful program evaluation.
•
Organize professional development events focusing on program evaluation.
•
Facilitate the establishment of professional networks supporting program
evaluation efforts.
N.B.: For examples of all of the above (and related resources), please
monitor the Foreign Language Program Evaluation Project (FLPEP) web site:
http://www.nflrc.hawaii.edu/evaluation
Curricular Reform, Student Outcomes, and Identity of Majors:
A Case Study of German Program Outcomes Assessment
Heidi Byrnes (byrnesh@georgetown.edu)
Peter C. Pfeiffer (pfeiffep@georgetown.edu)
Georgetown University, Department of German
I. Program description and overall purpose for evaluation
The Department’s curricular reform project “Developing Multiple Literacies” led to a unified set of educational goals for majors and non-majors alike. As a result, the question of the major’s identity, distinctiveness, and value arose. In order to evaluate this issue synchronically and diachronically, both current students and alumni are included in data gathering.
II. Intended uses, users, and evaluation questions
Intended uses
Primary uses of this evaluation are (a) to understand the distinctiveness
and value of the German major from the perspectives of multiple program
stakeholders; (b) to identify any needed changes, additions, and other
adjustments in the targeted outcomes of the German major; (c) to provide
a basis for curricular revision in the German studies degree program,
if needed.
Its secondary uses are (a) to increase awareness and engagement among
GUGD stakeholders about the nature and value of the German studies
degree and
the undergraduate curriculum; (b) to involve graduate students in work
on the GUGD program in order to facilitate their understanding of and
investment in the curriculum; (c) to provide an opportunity for teacher
development
through evaluation results; and (d) to sustain evaluation practices and
awareness of GUGD efforts within the Georgetown institutional context
and the broader national FL education context.
Intended users
Intended users are faculty; students; graduate students; alumni; administration.
Evaluation questions we posed are: (a) What are the characteristics of the major? (b) To what extent are they distinctive? (c) What values are attached to the major?
III. Methods, tools, and procedures
Three instruments were created for three distinct respondent groups: (a) web-based questionnaire with Likert-scale responses and open-ended responses (all students in German courses, Spring 2007); (b) focus groups (current majors at all levels, Spring 2007); (c) web-based questionnaire (methodology to be determined; alumni, AY 2007-08).
IV. Intended value added
Creation of an explicit basis for evaluation of the program and, as appropriate, for curricular, pedagogical, and administrative actions. Increased awareness internally; substantive representation of the program to outside stakeholders (administration; alumni; prospective students); modeling of FL program outcomes assessment to neighboring departments and the FL profession at large.
Note: Prof. Hiram Maxim was one of the lead PIs in the initial phase of
this project until summer 2007.
Summative Evaluation of 4th Year Hybrid Korean Course
Seungja Kim Choi (seungja.choi@yale.edu)Yale University, Center for Language Study
I. Program description and overall purpose for evaluation
The Korean language program is situated in the Department of East Asian Languages and Literatures along with the Japanese and Chinese programs at Yale University. Compelled by the unique situational characteristics of the Korean program, such as small enrollment, diverse backgrounds of heritage students, limited faculty resources (2 full time), and the need for more advanced language courses due to the new foreign language requirement at Yale, web-based materials were constructed for a 4th year on-line/in-class hybrid course. The course, K154 will be offered this fall for the first time. The temporary course approval calls for the evaluation of this course.
II. Intended uses, users, and evaluation questions
The intended use for this evaluation is permanent course approval, and
the primary intended users are Language Study Committee, New Course Committee,
and the department. The New Course Committee wants to know whether or
not this hybrid course is equal to a traditional course in terms of work
load and effectiveness (student learning outcomes), and whether or not
this course is different from the 3rd year course, K150. This evaluation
will focus on assessing the delivery and effectiveness of the course
in terms of student learning outcomes and work load by gathering information
from the students in the course.
III. Methods, tools, and procedures
In order to measure the effectiveness of the course, the information on student learning outcomes will be gathered through (a) pre and post course essays, and (b) pre and post reading comprehension test developed by the faculty members. The information on work load and effectiveness will be gathered via student questionnaire, and focus group interview.
IV. Intended value added
(a) We can find students’ learning strategies in online material
use, advantages and disadvantages of a hybrid course, and pedagogic strategies
of teaching through a hybrid course.
(b) If the result of this evaluation shows that a hybrid language course
at advanced level is comparable to or more effective than a traditional
class in terms of delivery and student learning outcomes, more advanced
literacy courses in other languages could be developed.
(c) Through detailed plans on student learning outcomes assessment, the
evaluator’s ability in course design and articulation, and teacher
training will grow.
(d) This evaluation might promote a program evaluation culture that is
systematic and sustaining, which contributes to curriculum and faculty
development at Yale.
Setting Benchmarks: First Year French Program
Hilary J. Fisher (hilaryj@uoregon.edu)Coordinator, First-year French
University of Oregon, Department of Romance Languages
I. Program description and overall purpose for evaluation
The undergraduate French program for first and second year is designed and implemented by two coordinators, but without the benefit of a departmental mission or set of goals for student achievement. The overall purpose is to begin the process of establishing standards for student performance, through determining what students are learning and what they are able to do with the language at the end of the first year of instruction.
II. Intended uses, users, and evaluation questions
Intended Uses
(a) Determine whether the current first-year French curriculum brings students
to the proficiency level recommended by the Oregon benchmarks.
(b) Plan revisions to the program based on the findings.
Expected users
First-year French teaching faculty, Graduate Teaching Fellows (GTFs) assigned
to first-year program, second-year French program director, and Department
Chair
Evaluation questions
(a) What parts of the current first-year curriculum teach to the topics
and functions recommended by the Oregon benchmarks?
(b) What parts are missing from the current curriculum?
(c) How are the gaps going to be addressed?
(d) What is redundant in the current curriculum? Are those redundancies
valuable?
(e) To what extent do the Oregon benchmarks reflect our ideas of what a
first-year curriculum should be?
(f) Following revisions to the program, how will we know students are achieving
the recommended benchmarks?
III. Methods, tools, and procedures
(a) Document analysis: First-year textbook, syllabus, Oregon benchmarks
(b) Map the current curriculum against the Oregon benchmarks
(c) E-mail questionnaire to first-year French language faculty and GTFs
on expected skills and knowledge for first-year learners
(d) Conduct student focus group on current program
(e) Hold meeting of first-year teaching faculty to discuss findings and
plan revisions
IV. Intended value added
(a) Improved articulation between the first and second year French program
(b) Highlighted status and importance of a first-year language program
(c) Improved student placements to increase efficient instruction to proficiency
(d) A mission statement for the 2-year French language program, and goals
and objectives calculated to fulfill it
Using Program Evaluation to Make a Case for a New Spanish BA
Rafael Gómez (rafael_gomez@csumb.edu)María Zielina (maria_zielina@csumb.edu)
Frauke Loewensen (frauke_loewensen@csumb.edu)
California State University, Monterey Bay,
School of World Languages and Cultures, Spanish Program
I. Program description and overall purpose for evaluation
Currently, the Spanish program is housed in the School of World Languages
and Cultures at California State University Monterey Bay and offers a bachelor’s
degree in World Languages and Cultures. The purpose of the evaluation project
is to add value to the existing program by collecting evidence in support
of the creation a bachelor’s degree in Spanish.
II. Intended uses, users, and evaluation questions
The evaluation will be used in order to gain approval of the “Permission to Plan” document, a required first step in the creation of a new program at our university. The intended users include department faculty, Dean, Provost, Academic Senate, students and alumni, as well as the community. The evaluation questions to be addressed include What value does the Spanish BA program add to the university? What evidence exists that there is a need for the BA in Spanish? In what ways will the Spanish BA respond to the vision of the university?
III. Methods, tools, and procedures
The evaluation consists of three phases: data collection, data interpretation, and presentation of findings. Raw data will come from document review/analysis and surveys. The latter include phone interviews, web-based questionnaires, and focus groups and will request input from other CSU FL chairs as well as stakeholders (Department faculty, students and alumni, community). The survey questions will be pilot-tested. Document review includes statistics (such as enrolment data) as well as documents published by our campus and other comparable institutions about program description, learning outcomes, curriculum, and the university’s vision/mission. Data interpretation will be conducted in the context of a Strengths Weaknesses Opportunities and Threats analysis, i.e. examining the strengths and weaknesses of the suggested degree program, as well as opportunities and threats it presents to the existing structure. The last phase, the presentation of findings, is the completion of the “Permission to Plan” document.
IV. Intended value added
This project will provide decision makers with pertinent data that will help them make an informed decision about the suggested program change. It will foster a culture of evaluation at our institution, facilitate the exchange of ideas between different stakeholders, and respond to the vision of our institution.
Student Learning Outcomes Assessment
Antonio Grau Sempere (ag87@evansville.edu)University of Evansville, Department of Foreign Languages
I. Program description and overall purpose for evaluation
We have different programs in the Department of Foreign Languages at University
of Evansville: there are programs that lead to a major degree (Spanish,
French, German) and programs that offer only minor degrees (Japanese, Russian).
These programs are part of a bigger program: Foreign Languages, which has
10 full time faculty members and 2 part time adjuncts. In our program, students
are required to complete certain coursework requirements (including language,
literature, culture, civilization and linguistics) and to complete at least
one semester (or eight credits of coursework) of study abroad. In their
last semester before graduation, students are also asked to complete an
exit exam that tests their reading and listening comprehension as well as
their oral and written expression. Other components of out program include
our mission statement and objectives that focus on preparing students for
the personal and professional challenges of a multicultural society and
a global marketplace and a list of learning outcomes that our students are
expected to achieve at graduation. In a nutshell our students are expected
to understand, analyze and speak the language and to know about the culture,
literature, history in which the language is a part of. The purpose of evaluation
is the improvement of the quality of our program.
II. Intended uses, users, and evaluation questions
The intended users are faculty, students, and the Dean. The intended use
is to improve our program by redesigning our curriculum, changing our
teaching methodology, etc. The Dean may base any decisions (monetary,
related to new faculty and new classes, etc.) related to our program
based on the information produced by the evaluation. Students may take
any necessary steps (additional classes, more time studying abroad, etc.)
in case the student does not reach the intended level. Prospective students
may base their decision on the success of our former students. The following
are the questions for evaluation:
Q1: To what degree do our students graduate with the basic knowledge we
want them to have?
Q2: To what degree are we providing them with the necessary resources for
them to graduate with the basic knowledge we want them to have?
Q3: To what degree are our students able to compete with other students
coming out of similar programs?
III. Methods, tools, and procedures
Q1: Structured interviews to be carried out in the student’s junior
year. The student’s advisor and at least one more faculty member
of UEFL program are present during the interview. The interviews are designed
to elicit the student’s attitudes and self-evaluation of their own
learning experience. The portfolio is assessed in their senior year possibly
during the capstone course. The portfolio will include a variety or materials:
(a) a self-reflection written piece that the student will have to present
to his/her advisor and at least two more faculty members of UEFL program.
This essay will have to address the student’s learning experience
at UE and comments on the papers included in his/her portfolio (b) at least
three term papers dealing with literature, linguistics and culture/history/current
events (c) any other additional piece of writing (term paper, self-reflection
piece, essay, creative writing, publication, etc.) that the student thinks
it is important to include.
Q2: Analysis of assessment results. The pattern in students’ results will determine the outcomes that are receiving enough attention in our curriculum and the ones that need more support in terms of curriculum rearrangement.
Q3: Unstructured interviews with alumni and chairs of departments of other institutions our size.
IV. Intended value added
I consider the value of this project has to do with the definition I gave
this assessment project itself: the ongoing process of reflecting upon
what we do in our program, based on systematic and objective evidence,
promoting internal (among faculty and students) and external (administrators)
communication, information and education; and judge the effectiveness
of our program to better determine its development and improvement.
Program Evaluation with Respect to State-Required Outcome Performance for Certification
Susan Knight (knigh1sm@cmich.edu)
Central Michigan University, Department of Foreign Languages, Literatures
and Cultures
I. Program description and overall purpose for evaluation
The Department of Foreign Languages, Literatures, and Cultures of Central
Michigan University offers a B.S. in Education for majors and minors
of French, German,
and Spanish. All FL language majors and minors on teaching degrees are presently
required to pass a Simulated Oral Proficiency Test at the Intermediate-High
level of the ACTFL scale before entering the FL methods course. This is an
institution test, created and assessed by the 24 members of the FL department,
some of whom have been formally, though minimally trained several years ago.
It has become increasingly important to evaluate the effectiveness of our
program’s speaking outcomes, especially since the State of Michigan
has changed the FL teacher certification policy, and will next year require
performance at the Advanced-Low (AL) level of the ACTFL OPI, a level also
required for NCATE certification
The purposes of the evaluation project are (a) to determine how well the FL Department is preparing its majors and minors in education to pass the new state certification criteria, and (b) to determine what changes are needed to help students achieve that level.
II. Intended uses, users, and evaluation questions
Intended users are majors and minors in program and faculty of FL department.
To a lesser degree, they include the Department of Education as well
as high school programs and the future students these people will be
teaching. Intended
uses center upon improving (a) the certification percentage of graduating
seniors, (b) the general FL program, (c) student satisfaction with
program, and (d)
student recruitment into the FL program. The following evaluation questions
are posed:
Q1: Can graduating FL education majors and minors pass the OPI at the Advanced-Low
level?
Q2: Where in the program are students getting the opportunity to develop
the competence needed to pass this test?
Q3: What program changes are needed
to assure this level of proficiency upon
graduation?
III. Methods, tools, and procedures:
Question 1: Can students pass the OPI at the Advanced-Low level?
(a) Analysis and examination of existing documents and reports. How many
students in the past have not passed parts of the Department’s SOPI?
How many tries? What parts of the test have they passed and not passed?
What were examiner
comments (error analysis)?
(b) ACTFL OPI. Graduating seniors will take the OPI to determine their
level. Results will be compared to how they scored on the institutions’ SOPI.
Question 2: Where in the program are opportunities to develop the competence
needed to pass this test?
(a) Focus Groups (seniors) Sample questions: What classes
helped them? What were tasks in these classes? Did they attend conversation
hours, etc? What
do they do outside of class to increase proficiency? Have they studied
abroad? How many hours of contact do they have with TL each day? How
did they prepare
for the SOPI? Suggestions for improvement?
(b) Questionnaire followed by Meeting (faculty) to determine
how proficiency is addressed in various classes (syllabi, materials,
in and out of class tasks,
methods of evaluation)
(c) SOPI at 200, 300 and 400 level for sample student
groups.
(d) Samples of student oral production in various required
courses at different levels evaluated by faculty.
Question 3: What changes are needed to insure this level of proficiency?
(a) Use of previously discussed Senior Focus Group and Faculty Discussion group
(b) Survey (of alumni now teaching) Possible questions:
What helped prepare you to teach in the TL and specifically to speak
proficiently? What would you
suggest to improve the program?
(c) Analysis and examination of existing documents and reports. Examine
syllabi to find out how course content and requirements relate to proficiency
standards
IV. Intended value added
This will not only increase articulation between members of the faculty
regarding overall student learning outcomes and how to achieve them,
but also lead to
a better articulated program as a whole. It will also allow possible
opportunities for faculty presentations and research publications.
Student Satisfaction Survey to Measure Students’ Attitude towards Their Learning at Different Stages of Their German Studies
Annette Kym (akym@hunter.cuny.edu)Hunter College, CUNY, German Department
I. Program description and overall purpose for evaluation
Hunter College, with an enrollment of approx. 20,000 students, is part
of the City University of New York. The German Department is a freestanding
department with 5 FT faculty members, 1 TA and a number of adjuncts. The
average enrollment per semester is between 360 – 400 students, appr.
28 – 30% of those students are in upper-level courses taught in the
target language. Hunter College has a 4-semester language requirement The
German major requires 24 credits, the minor 12 credits.
We want to evaluate student satisfaction at various stages of our program. In this process, we want to find out what the value of learning outcomes is to students finishing the language requirement, and to students finishing upper-level courses. In addition, we want to find out how effective the department’s teaching approach is as judged by the students.
II. Intended uses, users, and evaluation questions
Intended uses on the departmental level
By engaging in this evaluation, we want to find evidence on students’ perceptions
of the value of their language studies. The information will help the department
to take actions to improve the program. It will also be used in discussions
with the administration about resources allocated to the department.
Intended uses on the college level
We want to use the data in the broad discussion about the General Education
Requirement, which is currently under review. The systematically collected
data will produce evidence of the value of the FL requirement as perceived
by the students. Furthermore, the college is undergoing accreditation
by the Middle States Association and this type of information can be
used for that purpose.
III. Methods, tools, and procedures
We plan to collect the data through focus groups with students and student surveys. We will run a pilot project in the fall semester 2007, and conduct the focus groups and administer the finalized survey in spring 2008. The data will be analyzed over summer 2008 (Middle States Review will take place in Fall 2008).
IV. Intended value added
The German Department has engaged in outcomes assessment for a number
of years and we have data about student learning outcomes. However, we
have not collected data from the students’ point of view and measured
students’ perception about their learning outcomes systematically.
Getting data on their perceptions will fill this gap, and in conjunction
with other indicators, will help us to improve the program and assist us
in long-term program planning.
Evaluation of a Spanish Pilot Project in the Center for Languages and Cultures
Beth Landers (landersb@umsl.edu)Director of the Center for Languages and Cultures,
University of Missouri-St. Louis
I. Program description and overall purpose for evaluation
The Center for Languages and Cultures is located within the department
of Anthropology and Languages at the University of Missouri-St. Louis.
UMSL is an urban,
public research university with high numbers of non-traditional students
who commute to campus. The Center is responsible for elementary and intermediate-level
language courses in nine languages (Spanish, French, German, Japanese, Chinese,
Modern Greek, Arabic, Latin and Ancient Greek) as well as ESL. Approximately
40% of all UMSL undergraduates have a three-semester foreign language requirement
(all students who earn BA’s, as well as a few select other degrees).
The Center serves these students, and also endeavors to retain students beyond
the three-semester required sequence, preparing them to major or minor in
language.
The specific component of the program to be evaluated is a Spanish Pilot Project initiated and funded by the College of Arts and Sciences as part of an initiative to improve all required classes in Math, Foreign Languages, and English. This particular project focused on ways to improve the oral and written proficiency of students enrolled in the first three semesters of language courses (the required sequence). At its inception, no provision was made to evaluate the pilot project but it was intended to inform curricular changes in all lower-division classes across language sections.
II. Intended uses, users, and evaluation questions
This evaluation will identify best practices from the project that can
be generalized and applied to all sections of elementary French, German
and Spanish. The users will thus be the Director of the Center and the
thirteen instructors in these sections. Specific questions to be asked
are: What did the project do that was different from the past? What was
innovative? What was traditional? What worked well and why? What did
the teachers like? What did the students like? How did the Pilot Project
affect the oral and written proficiency levels of students? How did the
Pilot Project affect student success rates? Did the Pilot Project affect
student motivation?
III. Methods, tools, and procedures
In order to answer these questions we will use a variety of methods: 1)
a questionnaire distributed to instructors 2) a focus group of the three
instructors at the end of the project 3) analysis of video tapes of particularly
innovative class sessions 4) a student questionnaire distributed to 3rd
semester Spanish students in fall 2007, 5) comparison of OPI ratings
of the control group and the pilot project group in their third semester,
6) comparison of compositions on final exams with compositions from previous
years, 7) analysis of course materials developed over three semesters,
8) analysis of student course evaluations, 9) analysis of student attendance/absence
patterns 10) a report from the office of Institutional Research about
the pass/fail rate of students over the three semesters of the pilot
project, 11) a report from the OIR on how many students from this group
enrolled in Spanish 4 in spring 2008.
IV. Intended value added
This evaluation will allow us to target the best practices developed in
the Spanish Pilot Project and incorporate them in all basic language
classes in order to refresh teaching practices, bring a measure of standardization
to basic language instruction, and improve student learning outcomes.
We hope that doing so will increase the number of students who continue
their language study beyond the third semester. Ultimately, we hope to
encourage further investment in the Center for Languages and Cultures
by assuring the College that its initial investment in the Spanish Pilot
Project has been used in constructive ways.
Evaluating the Foreign Language Requirement at Duke University
Carolyn Lee (kslee@duke.edu)
Associate Professor of the Practice of Chinese
Ingeborg Walther (waltheri@duke.edu)
Associate Professor of the Practice
of German and incoming Assistant Dean of Trinity College
Duke University,
Foreign Languages Departments (Asian and African Languages and Literature,
Germanic Languages and Literature, Romance Studies, Slavic
and Eurasian Studies)
I. Program description and overall purpose for evaluation
In the year 2000, Duke University instituted a new curriculum for undergraduates
of Trinity College (the undergraduate college of the School of Arts and
Sciences), which includes a foreign language requirement. Whereas the
last several years have been devoted to addressing issues of implementation,
the Dean of the College is now in the process of evaluating the curriculum.
Most recently, a committee consisting of several language program directors
and the director of the assessment office has been asked to determine
a means of documenting student learning outcomes, for the purpose of
understanding the extent to which students are achieving the goals articulated
by the foreign language requirement.
II. Intended uses, users, and evaluation questions
The primary uses of the evaluation project are to demonstrate the merit
and value of the foreign language requirement by documenting student
learning outcomes, to chart student progress at various stages of their
foreign language learning, and to improve foreign language programs and
curricula. The information obtained from the evaluation project will
primarily be used by university administrators to document what students
are learning as a result of the foreign language requirement, and possibly
to more clearly identify and understand how individual language programs
prepare students to meet the foreign language requirement goals established
by the university. The information gathered will be used by individual
language program faculty to assess the extent to which their own programs
and curricula are preparing students to meet these goals, for the purposes
of engendering programmatic and curricular improvement. In particular,
we would like to investigate the following questions: 1) What levels
of language proficiency are students achieving in speaking, reading,
and writing? 2) What cultural knowledge, understandings, and perspectives
are students gaining at each level of the curriculum? 3) To what degree
do factors such as language program curricula, study abroad, heritage,
previous language learning experiences play a role in students’ learning
outcomes? 4) What are student perceptions of the university foreign language
requirement?
III. Methods, tools, and procedures
We will be using multiple methods to collect the information we need, including
SOPI or COPI, course embedded assessments, student portfolios, online
questionnaires, individual interviews and focus groups. We will most
likely collect this information in stages: using the fall semester of
2007 for planning and piloting, the spring semester of 2008 for initial
data collection, and the following academic year for analysis and additional
data collection, as informed by decisions based on collected data.
IV. Intended value added
We imagine that the very process of engaging in the evaluation project
will help individual language programs address issues of curricular goals
and coherence, faculty development, and articulation with other programs
and departments across the university. We expect that the project will
help communicate more precisely to the outside world (administrators,
faculty, students, parents, communities) just what it is our foreign
language programs do and what students learn. This may enhance the profile
of foreign language programs within the university and beyond as an integral
part of the humanities in a liberal arts education. And finally, we hope
that this project will contribute to the growing research in the field
of foreign language education and program evaluation.
Assessment of Student Learning Outcomes for the Spanish Major
Judith E. Liskin-Gasparro (judith-liskin-gasparro@uiowa.edu)
University of Iowa, Department of Spanish and Portuguese
I. Program description and overall purpose for evaluation
The undergraduate major in Spanish consists of 12 upper-division courses:
1 course in each of 4 core areas (Peninsular literature, Spanish American
literature, Culture, and Linguistics) and 8 elective courses. The purpose
of evaluation is to provide evidence that will inform faculty discussion
about the effectiveness of the curriculum and, thus, serve as the basis
for making changes to enhance and improve student learning. The immediate
impetus for the evaluation is the university-wide mandate to formulate
an outcomes assessment plan for the major for the upcoming reaccreditation
cycle.
II. Intended uses, users, and evaluation questions
Intended uses
(a) Make changes in the curriculum (types of courses, content of existing
courses)
(b) Increase number of/access to curricular components most highly valued
by students
(c) Review (and possibly adjust) the departmental hiring plan in light
of the evaluation
(d) Identify additional areas that would benefit from evaluative attention
(e) Provide evidence of program quality and departmental commitment to
improvement of student learning for internal and external proposals (e.g.,
faculty lines, funding)
Intended users: Faculty, students, administrators
Evaluation questions
Q1. How effective is our curriculum in teaching speaking skills?
Q2. How effective is our curriculum in teaching high-level writing skills?
Q3. Which components of the program do students value most highly as
contributing to their learning?
III. Methods, tools, and procedures
(a) Speaking (Q1): Locally designed interview procedure that focuses on
Intermediate Mid?Advanced. The outcomes assessment (OA) committee will
produce an interview protocol and train a small number of faculty in
the procedure. Speech samples will be designated as Meets Expectations
(Intermediate High), Exceeds Expectations (Advanced Low or higher), or
Does Not Meet Expectations (Intermediate Mid or lower).
(b) Content-based writing (Q2): Work samples (e.g., research papers)
embedded in senior-level courses will be assessed for the ability of
the writers
to produce in Spanish (i) coherent analyses and interpretations of Spanish/Latin
American literary texts or cultural phenomena, (ii) coherent summaries
and evaluations of linguistic phenomena and construction of linguistic
arguments, and (iii) coherent analyses of Spanish/Latin American films.
Students will create electronic portfolios that include 2 papers that
they believe represent their best work from 1 or more of their 3 highest-level
courses. A sample of these papers will be assessed by the OA committee
(Meets Expectations, Exceeds Expectations, and Does Not Meet Expectations)
according to a rubric designed by the committee for this purpose.
(c) Student perceptions of value of program (Q3): Students will write
for their portfolios a reflection essay, in which they will be asked
to reflect
on the aspects of the program that contributed most significantly to
growth in their (i) intellectual/academic knowledge and analytical
ability, (ii)
linguistic skills, and (iii) understanding of diverse cultural perspectives.
Focus groups will be held with a sample of graduating majors to address
the same topics. Reflection essays and focus group transcripts will
be analyzed for key themes related to points (i), (ii), and (iii).
IV. Intended value added
(a) Provide an evidentiary basis for understanding the program and how
it functions (evaluation) and to produce/implement a plan for change
(actions) to increase its effectiveness to enhance student learning.
(b) Communicate the value of our program to administrators, students, external
agencies, etc.
(c) Provide a site for research on teaching and learning in foreign language
programs.
Foreign Language Enrollment Growth Initiatives at Cal State L.A.
Sachiko Matsunaga (smatsun@calstatela.edu)
Chair of Department of Modern Languages and Literatures,
California State University, Los Angeles
I. Program description and overall purpose for evaluation
Our Foreign Language Program, which belongs to the College of Arts and Letters,
consists of MA programs in French and Spanish, BA programs in Chinese, French,
Japanese, and Spanish, and Subject Matter Preparation programs in French,
Japanese, and Spanish, as well as elementary level courses in Arabic, Italian,
Korean, and Portuguese. The main components that constitute our Foreign
Language Program are: courses, teaching, and students in pedagogical arrangement;
staffing and finance (for instructors, space and equipment) in institutional
arrangement.
II. Intended uses, users, and evaluation questions
Intended users are: primarily full-time faculty (CHIN; FREN; JAPN; SPAN),
Department Chair, and Dean; secondarily part-time faculty, students,
other departments, senior administrators, local schools, and communities.
Intended uses are (a) faculty act on the evaluation results (e.g., modify
courses/programs; identify new practical opportunities); (b) the Department
Chair seeks support from the Dean, who decides whether to advocate our
FL Program to the senior administrators to fund new initiatives; (c) senior
administrators decide whether to increase funding for our growth initiatives;
and (d) local schools and communities decide whether to express support
for potential changes to be made based on the evaluation findings and our
plan.
The first question of evaluation is: What are other CSU campuses doing in serving and recruiting FL students? We ask this question in order to know why our overall enrollment has declined (our SFR is lower than the average CSU SFR). Possible reasons could be environmental, institutional, and/or programmatic. The findings will help us offer explanations to the administrators. The second question of evaluation is: In what areas can we expect enrollment growth? We ask this question in order to know what actions we need to take in order to increase FL enrollments on our campus. The findings will be used to request support from the administrators.
III. Methods, tools, and procedures
Methods that we will employ are: (a) Internet search for factual information
about other CSU campuses; (b) questionnaire distributed to all CSU FL
Department Chairs electronically to find out what they are doing in serving
and recruiting students; (c) faculty meeting with the Department Chair
as a facilitator to come up with areas for growth; (d) focus group discussions
to investigate whether the areas for growth identified by faculty match
the needs of learners (current majors and non-majors as well as alumni),
local schools, and communities. Responses on the CSU survey will be collected
electronically, and (follow-up) phone interviews will be conducted if
necessary. We will use Strengths Weaknesses Opportunities and Threats
analysis to analyze all data, and present the results to the Dean and
senior administrators in order to convincingly communicate to them the
importance of the role that we play within the University and the necessity
of funding for the identified growth (with supporting evidence gathered
from all stakeholders) to happen.
IV. Intended value added
If we successfully gain support from the administrators with these initiatives,
we will be able to not only strengthen our existing programs for our
majors with new hires, more space, and better equipment, but also further
diversify our course offerings for non-majors. In addition, the role
that we play within the University will be raised, possibly to the extent
that the College or University might consider implementing a foreign
language requirement. Furthermore, it will prepare the Department well
for the program review that we need to start conducting 18 months from
now.
Evaluating the Modern Languages and Linguistics Department at UMBC
Ana Oskoz (aoskoz@umbc.edu)
Assistant Professor of Spanish,
University of Maryland Baltimore County, Modern Languages and Linguistics
department
I. Program description and overall purpose for evaluation
The Modern Languages and Linguistics (MLL) department at the University
Maryland Baltimore County (UMBC) emphasizes communication across boundaries.
The MLL program has been praised for its innovative and interdisciplinary
nature, but since 1985, the program has expanded in both scope and amount
of students. Recently, the faculty started re-examining the curriculum
for articulation purposes. Still, there is a need to frame the MLL curriculum
within the university mission statement, and to find means of documenting
learning outcomes that fit with the department and the university goals.
II. Intended uses, users, and evaluation questions
Intended uses
(a) To examine the extent to which MLL department’s (and each language
area) mission statement and goals fit with the university mission statement
and goals.
(b) To establish a common understanding in the MLL department (and in each
language area) of “knowledge,” “skills,” and “dispositions.”
(c) To establish expected student learning outcomes in the MLL department
(and in each language area).
Intended users
The intended users are faculty, current and prospective students, high
schools, and administration.
Evaluation questions
(a) How do the MLL department’s objectives fit with the University
mission?
(b) How do the Spanish area objectives fit with the MLL department mission?
(c) What is the common understanding of “knowledge,” “skills,” and “dispositions” that
MLL students need?
(d) How does the curricula (and specific classes, curricula, syllabi) in
the Spanish area provide students with the knowledge and skills needed
to achieve the program objectives?
(e) The fifth question is divided into 2 questions:
1. What are the MLL department student outcomes regarding writing, speaking
and culture?
2. How does the Spanish area determine whether Spanish area students
have achieved expected program outcomes (writing, speaking and culture)?
(f) How well does the program communicate program goals to the administration, other departments at UMBC and students (current and prospective)?
III. Methods, tools, and procedures
The department will used different methods/tools to collect information
needed such as (a) document analysis to understand alignment of mission
statements, (b) discussions with faculty to define relevant concepts,
(c) questionnaire to alumni to gain information regarding the validity
of the program mission statement and perceived program value, (d) foci
group with students to elicit information regarding program perceptions,
program components (speaking, writing, culture, etc.), and relationship
with faculty/program, (e) final papers and class discussions to observe
accomplishment of culture learner outcomes, and (f) short questionnaires
to administration and other departments to gain information regarding
professional relationship with other departments, and administration.
IV. Intended value added
The intended value added of the evaluation is to increase awareness of
how the MLL department addresses issues of curricular goals and coherence,
and articulation with other programs and departments across the university.
The evaluation will also emphasize the MLL program identity to outside
stakeholders (administration; alumni; prospective students, etc.), and
it will highlight its specific contributions to the students’ educational
experience within and beyond UMBC.
Tailoring Language Majors’ Year Abroad Experience to Improve Outcomes
Violeta Ramsay (vramsay@linfield.edu)
Chris Keaveney (ckeaven@linfield.edu)
Co-Chairs of Department of Modern Languages,
Linfield College
I. Program description and overall purpose for evaluation
The Department of Modern Languages at Linfield College offers minors and majors in French, German, Japanese, and Spanish (as well as first year courses in Chinese and Latin). The department has ten full-time faculty and 1-3 adjunct instructors per year. Language requirement for a B.A. degree is one year. The purpose of our evaluation is to improve the level of achievement of all majors.
II. Intended uses, users, and evaluation questions
The intended uses of evaluation are (a) to make sure all language majors have the same opportunities to reach the goals of the major; and (b) to improve students’ year-abroad experience. Our evaluation question we have posed is: What aspects of student’s preparation, the year abroad experience or the design of the capstone seminar cause wide disparities in levels of achievement among language majors?
III. Methods, tools, and procedures
These are the steps we take in the evaluation process:
(a) Design a manual for majors that contains information on all stages
of the major program
(b) Select programs that best match our program and goals; establish communication
with those programs, and ask students to select among them.
(c) Collect data from students to establish a basis for the observation
of development: This includes samples of written work and an oral proficiency
interview (OPI) before students leave.
(d) Increase contact with and supervision of students’ work while
they’re abroad, creating a one-credit course to make it possible.
The course calls for the following:
• Students samples of papers and two essays
• A second OPI by phone
• Initial work on senior project: choosing topic, making outline, collecting
data
• A third OPI upon return to home campus
(e) Make changes in capstone seminar to allow for more contact with advisors as well as more time, and guidance during the writing, revisions and presentation of senior project.
The following timeline guides the procedures
(a) Majors manual is given to students when they declare a major and reviewed
before departure
(b) Pre-departure data collection takes place during students’ 4th
semester
(c) Data collection from abroad takes place during 5th and 6th semester
(d) Guidance in capstone seminar takes place during students’ 7th
semester
IV. Intended value added
Improvement in the communication with and guidance for students during
their year abroad (most critical year for the major). The junior year experience
can be tailored to better suit the rigor and quality of our program at
home.
Preparation for Department Evaluation:
Building Faculty Capacity in Self-Evaluation
Montserrat Reguant (mreguant@msmc.la.edu)
Mount Saint Mary’s College, Los Angeles, Department of Language and Culture
I. Brief description of the program and overall purpose for evaluation
To prepare the Department of Language and Culture (LC) at Mount Saint Mary’s
College to respond to the cyclic evaluation required by the institution,
and at the same time to build faculty capacity in self-evaluation in order
to improve the effectiveness of the whole department.
II. Intended uses, users, and evaluation questions
The intended use of the evaluation activity is to improve the department
as a whole over time, as preparation for the evaluation of department
in spring of 2008. The intended users are faculty members of the Language
and Culture Department and Academic Vice-President-Provost.
Evaluation question: How can the Department of LC self-improve over time, in preparation for the departmental evaluation in 2008 and build faculty capacity in self-evaluation?
III. Methods, tools, and procedures
We are going to utilize self-assessment, using given indicators by the
Curriculum Committee on department goals, curriculum, faculty, and student
assessment, and institution. Surveys have already been prepared. They
will be distributed to the faculty of the LC department, filled out,
and collected during the first department meeting of 2007-2008 academic
year, August 27, 2007. Data will be analyzed in the following three weeks
and action for improvement will start before the end of the fall semester.
The following is a selection of indicators included in the surveys:
(a) Department goals: Do they address the needs of the students, the College Mission or strategic planning, long term planning, student assessment, and student retention?
(b) The Curriculum:
1. How does it support the department goals?
2. How does it meet the needs of majors and minors?
3. How does it compare to that of a similar institution?
(c) Faculty: Commitment to teaching, strong collegial environment, innovation in pedagogy, quality in course content, indicators of student success, on-going effort to strengthen the department, assessment of the department’s curriculum, integration of full-time and adjunct faculty, assessment of the department’s curriculum, assessment of the quality of the major
(d) Student Assessment:
1. The department objectives and assessment methods are aligned
2. Methods of assessment across sections
3. Embedded techniques of assessment across sections and/or courses
4. Student feedback is used in regular basis to assess the efficacy of
student assessment in
the department A variety of methods are used as part of student assessment
5. Use of at least one type of student-driven student assessment method
(e.g., electronic portfolio, senior capstone project, etc.)
(e) The Institution: Library offerings and services, tech resources and support, student support services, faculty support services, professional development and other enriching opportunities, equipment and facilities, and balance of full-time and adjunct faculty.
IV. Intended value added
This project will expand the culture of evaluation preparation for departmental
self-evaluation as an improvement tool. This self-reflection for departmental
improvement will ensure we are well prepared for the external evaluation
and at the same time we hope to be able to build faculty capacity in
self- evaluation to enhance student learning.
Development and Evaluation of Arabic Teacher Development Program
Martha Schulte-Nafeh (marthas@u.arizona.edu)
University of Arizona,
Center of Middle Eastern Studies and Department of Near Eastern Studies
I. Program description and overall purpose for evaluation
The Arabic Graduate Teaching Assistants (GTAs) have all attended the
same teacher training workshop in which a particular “vision” of
best language teaching practices was presented and discussed. The purpose
of the professional development program is to guide the GTAs to a fuller
understanding of those practices and to increase their ability to implement
them. The GTAs will engage in 6 teacher development activities: (a) participation
and reflection on lesson planning meetings, (b) reflection on their own
planning and teaching, (c) observation of colleagues, (d) videotaping
and reflective viewing of their own teaching, (e) seeking formative student
feedback on their teaching, and (f) being observed by the coordinator.
The teacher development program is being developed “evaluatively”;
That is, by considering evaluation questions throughout the planning
process and by including an explicit plan for formative and summative
evaluation of the program. The evaluation component of the plan is described
below:
II. Intended uses, users, and evaluation questions
Intended uses
(a) To determine and document the success of the program in developing
GTAs understanding of and ability to deliver language instruction that
is in line with the vision.
(b) To identify areas of weakness/possible improvement of the plan from
all stakeholders’ perspectives in order to further develop the plan.
(c) To inform the development of the vision itself as well as the way that
it is taught to new GTAs.
Intended users
The Arabic GTAs, The ME language coordinator, the director of the Center
for Middle Eastern Studies and the head of the Near Eastern Studies Department
Evaluation questions
(a) Does the development plan lead to increased teaching of Arabic based
on the identified good teaching practices?
(b) Does the development plan lead to increased sense of agency?
(c) Does the development plan lead to an increase in reflective language
teaching?
(d) Which of the activities in the plan are most useful and why?
(e) Is the development plan meeting the users (GTAs) needs?
III. Methods, tools, and procedures
A variety of data sources will be used to answer the questions including: qualitative analysis of GTAs written reflection on their lesson planning and teaching; questionnaires for GTAs in the middle of the training period and at the end; faculty classroom observations of GTAs teaching; student questionnaires
IV. Intended value added:
The evaluation process will provide information from students, GTAs, and
other faculty which can contribute to the articulation and development
of a program wide understanding of good teaching practices.
USCB Spanish Evaluation: “From Growing Pains to Student Gains”
Babet Villena-Alvarez (jivillen@gwm.sc.edu)
University of South Carolina Beaufort, Foreign Language
I. Program description and overall purpose for evaluation
(a) Student Enrollment (approximate): Certificate (8 courses): 15 students;
Minors (9 courses): 8 students; Majors (10 courses for Heritage track
or 16 for General major): 13 students
(b) Institutional Background: USCB—newest baccalaureate degree-granting
institution of state; 1,200 students in state’s fastest growing region.
(c) Program Description: Spanish degree is 9th degree approved by the South
Carolina State Commission on Higher Education (SC-CHE) in January 2006.
The overall purpose is “to evaluate how well the Spanish curriculum at USCB prepares its students in the major, minor and certificate programs.” This evaluation is summative (to judge and value) and formative (to develop and improve).
II. Intended uses, users, and evaluation questions
Intended uses
(a) Improve Program (Quality of Spanish curriculum)
(b) Conduct Outcomes Assessment and Institutional Effectiveness (Quality
of faculty)
(c) Promote Student Advancement, Retention, and Recruitment (Quality of
students)
(d) Support other university programs (Quality of university curriculum).
Intended users
Spanish Faculty (Full-time & Part -time), Office of Institutional Effectiveness,
Office of Academic Affairs, Spanish students, community (local businesses & offices).
Evaluation questions
Overall question: How well does the Spanish curriculum prepare its students
in the major, minor and certificate programs?
(a) What do major/minor/certificate students need to know to communicate
effectively in Spanish in their career choice?
(b) With respect to cultural competency, how capable are the students in
distinguishing Spanish/Iberian cultures from the Hispanic/Latin American
cultures?
(c) With respect to the major/minor/certificate in Spanish, what are the
students’ goals and motivations?
(d) To what extent do students in the major/minor/certificate programs
participate and enhance their cultural knowledge through activities and
resources outside class (i.e., movies/audiovisual materials in the target
language, club activities, international festivals and other events, and
volunteer work where the target language is used)?
III. Methods, tools, and procedures
Method
Survey – electronic and/or paper (conduct a pre-program and a post-program
survey); exit interview with the Program Director or a Full-time Faculty;
tests – in-course; testimonials – at the end of the exit interview;
Institutional Effectiveness and Outcomes Assessment.
Tools
Survey instrument – two questionnaires; exit interview; list of
cultural content items relating to the specific items to be evaluated
that need
to be embedded and incorporated in specific course syllabi and tests;
form to be completed for testimonials; Institutional Effectiveness and
Outcomes
Assessment document for the Office of Institutional Effectiveness (For
accreditation purposes).
IV. Intended value added
The main directive from the USCB administration for the coming academic
year is to grow the Spanish Program. In the process, an effective evaluation
plan needs to be embedded in the growth process to ensure on-going curricular
improvement, to put in place faculty accountability, and to substantiate
student language proficiency.
Evaluation of the Arabic and Persian Summer Language Institute,
Georgetown University
Kassem Wahba (kmw@georgetown.edu)
Georgetown University, Department of Arabic and Islamic Studies
I. Program description and overall purpose for evaluation
In collaboration with the Department of Arabic and Islamic Studies, the
Summer School in the Department of Continuing Education offers an Arabic
language program within the Arabic & Persian Summer Institute. The
Arabic program is an undergraduate language program of intensive and non-intensive
courses which last for ten weeks. The courses offered include basic, intermediate
and advanced level Modern Standard Arabic and elementary classes in the
Iraqi, Levantine, and Egyptian Arabic dialect. The number of students enrolled
for Arabic and Persian is approximately 60 and 13, respectively. The students’ motivations
for taking Arabic include wanting to work for the government, wanting to
specialize in the Middle East, or fulfilling education requirements.
The overall purposes of evaluation are (a) to offer the administration information regarding the program’s curriculum structure and its impacts, and (b) to ascertain the effectiveness of the curriculum.
II. Intended uses, users, and evaluation questions
Intended uses
(a) To find out how well the current Arabic language curriculum meets students’ needs
(b) To identify what is missing in the curriculum and to propose a plan
to implement necessary changes for improving the curriculum
Intended users
Various administrators responsible for the program at the level of the
university, within the department of Arabic and Islamic Studies and the
Summer School of Continuing Education.
Evaluation questions
(a) What are the learners’ needs in studying Arabic?
(b) What are the current student learning outcomes?
(c) Is there a match between the outcomes and the learners’ needs?
(d) What changes are needed in order to better respond to learners’ needs?
III. Methods, tools, and procedures
Needs analysis will be carried out to gather information with regard to administrations, faculty, and learners’ perceptions of the students’ learning needs. The tools to be used are: interviews and a questionnaire.
IV. Intended value added
(a) To increase awareness among the administration and the faculty of
the potential need to modify the Arabic language curriculum to meet learners’ needs.
(b) To understand and improve the current program practice.
(c) To focus on the effectiveness of teaching and learning in terms of
learning outcomes.
Identifying a Nationally Accepted Tool to Assess Student Learning Outcomes
in the LCTL’s at the University of Florida
Ann Wehmeyer (wehmeyer@aall.ufl.edu)
Associate Professor and Chair
University of Florida, Department of African & Asian Languages & Literatures
I. Program description and overall purpose for evaluation
The goal of the program is twofold. First, we wish to determine whether
student outcomes in our foreign language programs are comparable to those
in external foreign language programs. Second, we want to ensure that training
capacities for nationally accepted proficiency testing are incorporated
into all of our language programs (Akan, Amharic, Arabic, Chinese, Hebrew,
Hindi, Japanese, Korean, Swahili, Vietnamese, Wolof, Xhosa, Yoruba).
II. Intended uses, users, and evaluation questions
The users of the evaluation program are department faculty, undergraduate and graduate students of language courses, area studies center directors, CIBER director, Dean of International Studies, and accreditation agencies. For faculty, the evaluation will serve to clarify our respective statements on student learning outcomes, and to identify a nationally recognized assessment tool which can be effectively applied to assess one or more of these outcomes. It will also enable us to incorporate into our language program features in common with external language programs. For students, the evaluation will serve to provide them with an opportunity to obtain nationally recognized language credentials. For administrators in related programs and centers, the evaluation will supply data that can be used to substantiate claims about the quality of our foreign language programs for purposes of grant application and reporting. For accreditation, the evaluation will serve to put in place an ongoing system of monitoring and assessment.
The overarching question underlying this evaluation program is to determine whether student outcomes in our language programs are comparable to those in external language programs. We will need to take the following steps in order to address this larger question: (a) What are the student learning outcomes in each of our respective language areas?, (b) What is the most commonly accepted assessment tool in each of the language areas?, (c) Is it feasible to apply this assessment tool to all students at selected levels of language? The answer to question (c) will require an investigation of fit to student learning outcomes, cost of materials, and cost of human resources investment in training and rating.
III. Methods, tools, and procedures
While languages that are part of a major have explicit student learning outcomes, those that are not yet part of a major lack formal statements in this regard. The language coordinator of each language unit will work with faculty in the unit to articulate, or revise, a formal statement of student learning outcomes. Secondly, each unit will determine which assessment tool among those currently available (e.g., for oral proficiency, ACTFL’s OPI, SOPI, COPI; for reading and writing proficiency, Oregon’s STAMP, etc.) is most commonly applied in their language area. In conjunction with the chair, each language unit will determine the costs of materials, and costs of human resources investment. Purchase of materials and training will commence along a timeline of feasibility. The selected assessment tool will be applied toward the end of the second year of language study, and also toward the end of the final semester of language study in those languages that are part of a major program. Results will be reported to all users.
IV. Intended value added
As faculty become knowledgeable about national assessment measures, their
ability to incorporate skills targeted by nationally recognized tests into
their respective language curricula will be enhanced, thereby bringing
our programs in the LCTLs more in line with the goals of programs in the
more commonly taught languages.
Use of Evaluation for Developing Russian Summer Abroad Program
Valentina Zaitseva, (vaz2@u.washington.edu)
Lecturer, University of Washington,
Department of Slavic Languages and Literatures
I. Program description and overall purpose for evaluation
The UW Department of Slavic Languages and Literatures has made a decision to offer a Russian summer abroad program in Sochi and to model it after the existing Czech summer program in Prague (set as Exploration Seminar, 3 weeks, 6 credits). The overall purpose of the evaluation project is to evaluate and inform the development of Russian Summer Program Abroad (Sochi).
II. Intended uses, users, and evaluation questions
The intended uses of this project are (a) to establish how the Russian Summer Program would contribute to the goals and missions of the Department; and (b) to evaluate how the Summer Program in Sochi would meet the learners’ needs and satisfy the requirements of Social Science Research Council (SSRC) funding for Russian Summer programs.
The intended users of the evaluation are chair, administrator and senior faculty, Russian summer program faculty; SSRC, intermediate and advanced UW students, graduate students, and faculty and administrators in Sochi University.
Evaluation questions
(f) What are pros and cons for setting a 4-week intensive language program
(45 contact hours equivalent of one quarter)? Which models for summer
program would better meet the students’ needs, departmental standards,
and SSRC requirements?
(g) What kind of curriculum would meet the set goals? How can we build
in special projects in under-/graduate research using local Sochi opportunities?
(h) How can local Sochi opportunities be used in task-based language teaching
or special socio-cultural projects curriculum?
(i) Which components of the Czech summer program in Prague and recruitment
strategies are transferable to Russian program in Sochi?
(j) How would Russian Summer courses fit curricularly into the overall
Slavic Department program? What implications for regular curriculum would
Summer program entail?
(k) What outcome assessment should be implemented in the end of the Russian
Summer program?
III. Methods, tools, and procedures
We are going to use learner needs analysis (through survey questionnaires and analysis of program requirements). We will collect pertinent data and perform a document analysis (UW program Abroad and other FL institutions websites, information about the Sochi University). Individual interviews, meetings and focus group discussions will help us to build consensus on length, intensity and curriculum of Russian summer program in Sochi. Informal preliminary meetings with faculty in Sochi in August, 2007 would outline the specific evaluative points to be incorporated into the next cycle of evaluative and program-building activities.
IV. Intended value added
While the development of the Russian Summer Program Abroad is in its very initial stage, the evaluative approach to program building helps to address a broad range of topics and issues in a practical and specific way, taking into consideration needs and improvement of the entire Slavic Department program.
Evaluating the Need for Formal Graduate L2 Teacher Training
Alessandro Zannirato (alessandro1@jhu.edu)
Johns Hopkins University,
Department of German and Romance Languages and Literatures
I. Brief description of the program and overall purpose for evaluation
The Italian section offers both majors and minors in Italian studies and
has a vibrant PhD program in Italian literature. Most of the doctoral
candidates are required to start teaching undergraduate language courses
right after being admitted into the PhD program. Many of them have limited
training in L2 teaching. Presently, the only training that the GTAs receive
takes place through weekly meetings with the Program Directors or coordinators
and through a reflection on classroom observations. It may be wondered
if this is sufficient, or whether some sort of more formalized training
may be advisable.
II. Intended uses, users, and evaluation questions
Intended uses and users
Through this evaluation, it is endeavored to: (a) evaluate the usefulness
of GTA formal training in L2 language teaching in order to improve the
quality of the undergraduate program; (b) reflect on the graduate curriculum
and weigh the possibility to include formal training in L2 language teaching;
and (c) verify if the investment needed to introduce formal L2 language
teacher training would be justifiable. The primary intended users are:
Language Program Directors, Section Heads and the Department Chair. Secondary
intended users would include the Dean’s Office and graduate students.
Evaluation questions
The evaluation questions that were asked are: (a) Is it possible to improve
the quality of our undergraduate courses through structured GTA training
in L2 teaching? (b) Can we make the GTAs’ academic experience a
better one? (c) Could we provide our graduate students with skills and
formal training to become more competitive in the job market? (d) Is
our department following nationally-recommended practices with regard
to graduate student teacher training?
III. Methods, tools, and procedures
Document analysis, focus groups, meetings and questionnaires will be used.
We will create an online questionnaire using Surveymonkey software, and
review the journal articles from the Chronicle of Higher Education. The
following are the steps we are going to take.
(a) Program directors
could review last year’s teaching observation
reports across the different languages, identify the repeating themes in
order to get an idea about the quality of GTA teaching and formulate recommendations.
(b) A focus group with one GTA from each language could be organized in
order to prepare a questionnaire to be distributed to all GTA, with the
objective to verify whether they are satisfied with their teaching abilities.
GTA responses may then be triangulated with program directors’ recommendations.
(c) Administer a survey to literature faculty members to verify their perceptions
of L2 language teacher training and its relevance in the graduate curriculum.
(d) Review recent job advertisements to ascertain whether formal training
in L2 language teaching is considered as a necessary or desirable qualification
by search committees.
(e) Identify a representative sample of peer institutions and review what
the role of L2 teacher training is in their curricula. Review FL professional
organizations’ recommendations (e.g. MLA/ACTFL etc.).
IV. Intended value added
This evaluation will hopefully contribute to a better understanding of
the role applied linguistics can play in today’s foreign language
departments.