Developing useful evaluation practices in college foreign language programs, 2007 NFLRC Summer Institute
* Following are short statements and descriptions of projects, all of which were produced by the participants in the NFLRC Summer institute. Note that participants will be adding to this information over the next year (2007~2008) as their projects progress. If you would like to print this page, please download the original booklet.[download pdf file]
• Introduction to the Summer
Institute
• Joint
statement on the value of evaluation
• Strategies
for culture change
1. Developing and improving
study abroad programs
- Russian:
University of Washington
- Multiple
languages: Linfield College
2. Effectiveness of innovation
- Korean:
Yale University
- Spanish:
University of Missouri St. Louis
3. Understanding program value
- German:
Georgetown University, Hunter College CUNY
4. Institutional program review
- Multiple
languages: Mount Saint Mary's College
5. Learner needs and curriculum
improvement
- Arabic
summer program: Georgetown University
- Spanish:
U of South Carolina Beaufort
6. Program alignment with
external benchmarks
- French:
University of Oregon
- Multiple
languages: Central Michigan University
7. Program development and
advocacy
- Spanish:
Cal. State University Monterey Bay
- Multiple
languages: Cal. State University Los Angeles
8. Assessing student learning
outcomes
- Spanish:
University of Iowa
- Multiple languages: Duke
University, University of Evansville,
University
of Florida, University of Maryland Baltimore
9. Teacher and GTA development
- Arabic:
University of Arizona
- Italian:
Johns Hopkins University
Introduction
From May 28 through June 6, 2007, 24 faculty members from a diversity
of college foreign language departments across the U.S. participated in
the 2007 Summer Institute of the National Foreign Language Resource
Center at the University of Hawai‘i. The focus of this
institute was Developing useful evaluation practices in college foreign
language programs. Sponsored by the NFLRC (http://www.nflrc.hawaii.edu),
in collaboration with John Norris’s federally funded Foreign
Language Program Evaluation Project (http://www.nflrc.hawaii.edu/evaluation),
the goal of the institute was to help college foreign language
administrators and faculty engage in useful, practical, and effective
program evaluations to meet a variety of purposes. A secondary goal was
to elevate the discourse about assessment and evaluation in higher
education, and to engender nationwide improvements in the contributions
made by foreign language education. In an era of zealous accountability
testing, and in light of inevitable changes in the educational and
societal roles being played by the foreign languages, the hope was to
develop and support a cadre of FL professionals to lead the way towards
a rationale and useful evaluation practice.
Participants in the institute surpassed all expectations. Working
through the Memorial Day holiday and into the weekend, they received
training in program evaluation concepts, models, and methods; they
translated and applied evaluation ideas to the very real challenges of
contemporary college FL education contexts; and they developed extended
plans for the use of evaluation in response to unique questions in
their own particular language programs. Above and beyond these planned
institute activities, participants also engaged in extended discussions
about the value of language education in the U.S., they articulated
fundamental tensions that define much of the FL teaching and learning
landscape, and they forged potential solutions based on sound
evaluative practices. They even adopted a name for themselves: the
Faculty Working Group on Foreign Language Program Evaluation.
In this booklet, the Faculty Working Group on Foreign Language Program Evaluation summarize three immediate outcomes of their work during the institute. First, they provide a statement on the Value of Evaluative Thinking and Action in Foreign Language Programs. Second, they offer broad Strategies for Culture Change in Program Evaluation. Third, they present one-page overviews of their program-specific evaluation work, as one way of demonstrating the range of possible contributions to be made by evaluation in college FL education. It is their hope that these collected ideas will serve as a springboard for dialogue and action within the FL disciplines.
On behalf of all of the summer institute’s organizers and facilitators, we congratulate the participants on their hard work and excellent achievements, and we look forward to sustained collaborations with them in the service of useful foreign language program evaluation.
John Norris, Director
Yukiko Watanabe, Assistant Director
NFLRC Summer Institute 2007
The Value of Evaluative Thinking and Action in Foreign Language Programs
A joint statement by the Faculty Working Group on Foreign Language Program Evaluation
This statement is the result of discussions among the participants in the NFLRC Summer Institute 2007. The intent is to articulate the value of evaluative thinking and action to foreign language education.
Evaluative thinking and action provides a framework for discussion in
programs or departments about fundamental questions of program
effectiveness. These discussions can have a democratizing and unifying
effect—democratizing because all voices are heard, and
unifying because the process leads to communication and consensus
building. Collaborative discussion and action that involves all
stakeholders results in a heightened commitment of all participants to
the vitality of the program, thus contributing to a sense of academic
community.
The evaluation process allows faculty members to understand the program
as a whole and to articulate to themselves and others what they want
students to achieve in the areas of knowledge, skills, and
dispositions. By identifying strengths and weaknesses, they formulate a
plan (or plans) of action to increase program effectiveness and
maximize student learning. The goal is to make the learning process
more efficient and to create a well-articulated curriculum that is
responsive to changing circumstances, all within a cyclical process of
innovation and evaluation.
Evaluative thinking and action has further benefits. It enables
departments to address in action-oriented ways common problems at the
program level, such as low enrollments in some languages, attrition at
various levels, and difficulties in the curricular transition from
lower-division to upper-division courses. It offers opportunities for
individual faculty members to engage in professional development
activities, such as scholarship in teaching and learning and improving
teaching practices through ongoing reflection. It can increase
communication across departments, leading to cross-pollination between
disciplines and opportunities for collaboration with colleagues on
evaluation projects, as well as professional activities in other areas.
Beyond the department level, evaluative thinking and action enables
faculty members to enhance the profile of their program or department
within the institution by establishing themselves as leaders in
evaluation initiatives and showcasing the accomplishments of their
evaluation-related projects. Such leadership activities position the
program or department well in requests for support (e.g., funding,
faculty lines). Finally, the ability to demonstrate cycles of
innovation and evaluation empowers foreign language professionals,
enabling them to make a strong case for the unique contributions of
language studies in a pluralist and globalized world.
Strategies for Culture Change in Program Evaluation
A joint statement by the Faculty Working Group on Foreign Language Program Evaluation
This statement is the result of discussions among participants in the NFLRC Summer Institute 2007. The intent is to encourage the foreign language field to recognize program evaluation as indispensable for enhancing student learning and program quality, and to enable the field to articulate and demonstrate—internally and externally—the unique contributions of language studies in a pluralist and globalized world.
Strategies for changing perceptions of evaluation and enhancing value of evaluation
• Focus on program
improvement as a goal of program evaluation.
• Emphasize the usefulness of evaluation for: (1) student
learning, (2) program articulation, (3) departmental collaboration, and
(4) academic community.
• Highlight the public, participatory, and inclusive nature of
the evaluation process.
• Link evaluation goals to stated institutional priorities.
Strategies for encouraging faculty-led evaluation
• Build on program
information (curriculum, syllabi, final exams, papers, etc.) and
systematize evaluation work already conducted in the department.
• Lead institutional evaluation efforts by example; forge
alliances across the institution; draw on available institutional
resources.
• Appropriately recognize and incentivize evaluation work
within the department and the institution.
• Integrate evaluation into standard administrative,
curricular, and teaching practices.
• Pursue professional development opportunities and external
funding.
• Generate and showcase successful examples of evaluation.
Strategies for professional organizations to enhance useful evaluation
• Recognize and
disseminate successful models of program evaluation.
• Develop policy statements on useful program evaluation.
• Organize professional development events focusing on program
evaluation.
• Facilitate the establishment of professional networks
supporting program evaluation efforts.
N.B.: For examples of all of
the above (and related resources), please monitor the Foreign Language
Program Evaluation Project (FLPEP) web site:
http://www.nflrc.hawaii.edu/evaluation
Curricular
Reform, Student Outcomes, and Identity of Majors:
A Case Study of German Program Outcomes Assessment
I. Program description and overall purpose for evaluation
The Department’s curricular reform project “Developing Multiple Literacies” led to a unified set of educational goals for majors and non-majors alike. As a result, the question of the major’s identity, distinctiveness, and value arose. In order to evaluate this issue synchronically and diachronically, both current students and alumni are included in data gathering.
II. Intended uses, users, and evaluation questions
Intended
uses
Primary uses of this evaluation are (a) to understand the
distinctiveness and value of the German major from the perspectives of
multiple program stakeholders; (b) to identify any needed changes,
additions, and other adjustments in the targeted outcomes of the German
major; (c) to provide a basis for curricular revision in the German
studies degree program, if needed.
Its secondary uses are (a) to increase awareness and engagement among
GUGD stakeholders about the nature and value of the German studies
degree and the undergraduate curriculum; (b) to involve graduate
students in work on the GUGD program in order to facilitate their
understanding of and investment in the curriculum; (c) to provide an
opportunity for teacher development through evaluation results; and (d)
to sustain evaluation practices and awareness of GUGD efforts within
the Georgetown institutional context and the broader national FL
education context.
Intended
users
Intended users are faculty; students; graduate students; alumni;
administration.
Evaluation questions we posed are: (a) What are the characteristics of the major? (b) To what extent are they distinctive? (c) What values are attached to the major?
III. Methods, tools, and procedures
Three instruments were created for three distinct respondent groups: (a) web-based questionnaire with Likert-scale responses and open-ended responses (all students in German courses, Spring 2007); (b) focus groups (current majors at all levels, Spring 2007); (c) web-based questionnaire (methodology to be determined; alumni, AY 2007-08).
IV. Intended value added
Creation of an explicit basis for evaluation of the program and, as appropriate, for curricular, pedagogical, and administrative actions. Increased awareness internally; substantive representation of the program to outside stakeholders (administration; alumni; prospective students); modeling of FL program outcomes assessment to neighboring departments and the FL profession at large.
Note: Prof. Hiram Maxim was one of the lead PIs in the initial phase of
this project until summer 2007.
Summative Evaluation of 4th Year Hybrid Korean Course
Seungja Kim Choi (seungja.choi@yale.edu)Yale University, Center for Language Study
I. Program description and overall purpose for evaluation
The Korean language program is situated in the Department of East Asian Languages and Literatures along with the Japanese and Chinese programs at Yale University. Compelled by the unique situational characteristics of the Korean program, such as small enrollment, diverse backgrounds of heritage students, limited faculty resources (2 full time), and the need for more advanced language courses due to the new foreign language requirement at Yale, web-based materials were constructed for a 4th year on-line/in-class hybrid course. The course, K154 will be offered this fall for the first time. The temporary course approval calls for the evaluation of this course.
II.
Intended uses, users, and evaluation questions
The intended use for this evaluation is permanent course approval, and
the primary intended users are Language Study Committee, New Course
Committee, and the department. The New Course Committee wants to know
whether or not this hybrid course is equal to a traditional course in
terms of work load and effectiveness (student learning outcomes), and
whether or not this course is different from the 3rd year course, K150.
This evaluation will focus on assessing the delivery and effectiveness
of the course in terms of student learning outcomes and work load by
gathering information from the students in the course.
III. Methods, tools, and procedures
In order to measure the effectiveness of the course, the information on student learning outcomes will be gathered through (a) pre and post course essays, and (b) pre and post reading comprehension test developed by the faculty members. The information on work load and effectiveness will be gathered via student questionnaire, and focus group interview.
IV. Intended value added
(a) We can find
students’ learning strategies in online material use,
advantages and disadvantages of a hybrid course, and pedagogic
strategies of teaching through a hybrid course.
(b) If the result of this evaluation shows that a hybrid language
course at advanced level is comparable to or more effective than a
traditional class in terms of delivery and student learning outcomes,
more advanced literacy courses in other languages could be developed.
(c) Through detailed plans on student learning outcomes assessment, the
evaluator’s ability in course design and articulation, and
teacher training will grow.
(d) This evaluation might promote a program evaluation culture that is
systematic and sustaining, which contributes to curriculum and faculty
development at Yale.
Setting Benchmarks: First Year French Program
Hilary J. Fisher (hilaryj@uoregon.edu)Coordinator, First-year French
University of Oregon, Department of Romance Languages
I. Program description and
overall purpose for evaluation
The undergraduate French program for first and second year is designed and implemented by two coordinators, but without the benefit of a departmental mission or set of goals for student achievement. The overall purpose is to begin the process of establishing standards for student performance, through determining what students are learning and what they are able to do with the language at the end of the first year of instruction.
II.
Intended uses, users, and evaluation questions
Intended Uses
(a) Determine whether the current first-year French curriculum brings
students to the proficiency level recommended by the Oregon benchmarks.
(b) Plan revisions to the program based on the findings.
Expected
users
First-year French teaching faculty, Graduate Teaching Fellows (GTFs)
assigned to first-year program, second-year French program director,
and Department Chair
Evaluation
questions
(a) What parts of the current first-year curriculum teach to the topics
and functions recommended by the Oregon benchmarks?
(b) What parts are missing from the current curriculum?
(c) How are the gaps going to be addressed?
(d) What is redundant in the current curriculum? Are those redundancies
valuable?
(e) To what extent do the Oregon benchmarks reflect our ideas of what a
first-year curriculum should be?
(f) Following revisions to the program, how will we know students are
achieving the recommended benchmarks?
III.
Methods, tools, and procedures
(a) Document analysis:
First-year textbook, syllabus, Oregon benchmarks
(b) Map
the current curriculum
against the Oregon benchmarks
(c) E-mail questionnaire
to first-year French language faculty and GTFs on expected skills and
knowledge for first-year learners
(d) Conduct student focus
group on current program
(e) Hold meeting of first-year teaching faculty to
discuss findings and plan revisions
IV.
Intended value added
(a) Improved articulation between the first and second year French
program
(b) Highlighted status and importance of a first-year language program
(c) Improved student placements to increase efficient instruction to
proficiency
(d) A mission statement for the 2-year French language program, and
goals and objectives calculated to fulfill it
Using Program Evaluation to Make a Case for a New Spanish BA
Rafael Gómez (rafael_gomez@csumb.edu)María Zielina (maria_zielina@csumb.edu)
Frauke Loewensen (frauke_loewensen@csumb.edu)
California State University, Monterey Bay,
School of World Languages and Cultures, Spanish Program
I.
Program description and overall purpose for evaluation
Currently, the Spanish program is housed in the School of World
Languages and Cultures at California State University Monterey Bay and
offers a bachelor’s degree in World Languages and Cultures.
The purpose of the evaluation project is to add value to the existing
program by collecting evidence in support of the creation a
bachelor’s degree in Spanish.
II. Intended uses, users, and evaluation questions
The evaluation will be used in order to gain approval of the “Permission to Plan” document, a required first step in the creation of a new program at our university. The intended users include department faculty, Dean, Provost, Academic Senate, students and alumni, as well as the community. The evaluation questions to be addressed include What value does the Spanish BA program add to the university? What evidence exists that there is a need for the BA in Spanish? In what ways will the Spanish BA respond to the vision of the university?
III. Methods, tools, and procedures
The evaluation consists of three phases: data collection, data interpretation, and presentation of findings. Raw data will come from document review/analysis and surveys. The latter include phone interviews, web-based questionnaires, and focus groups and will request input from other CSU FL chairs as well as stakeholders (Department faculty, students and alumni, community). The survey questions will be pilot-tested. Document review includes statistics (such as enrolment data) as well as documents published by our campus and other comparable institutions about program description, learning outcomes, curriculum, and the university’s vision/mission. Data interpretation will be conducted in the context of a Strengths Weaknesses Opportunities and Threats analysis, i.e. examining the strengths and weaknesses of the suggested degree program, as well as opportunities and threats it presents to the existing structure. The last phase, the presentation of findings, is the completion of the “Permission to Plan” document.
IV. Intended value added
This project will provide decision makers with pertinent data that will help them make an informed decision about the suggested program change. It will foster a culture of evaluation at our institution, facilitate the exchange of ideas between different stakeholders, and respond to the vision of our institution.
Student Learning Outcomes Assessment
Antonio Grau Sempere (ag87@evansville.edu)University of Evansville, Department of Foreign Languages
I.
Program description and overall purpose for evaluation
We have different programs in the Department of Foreign Languages at
University of Evansville: there are programs that lead to a major
degree (Spanish, French, German) and programs that offer only minor
degrees (Japanese, Russian). These programs are part of a bigger
program: Foreign Languages, which has 10 full time faculty members and
2 part time adjuncts. In our program, students are required to complete
certain coursework requirements (including language, literature,
culture, civilization and linguistics) and to complete at least one
semester (or eight credits of coursework) of study abroad. In their
last semester before graduation, students are also asked to complete an
exit exam that tests their reading and listening comprehension as well
as their oral and written expression. Other components of out program
include our mission statement and objectives that focus on preparing
students for the personal and professional challenges of a
multicultural society and a global marketplace and a list of learning
outcomes that our students are expected to achieve at graduation. In a
nutshell our students are expected to understand, analyze and speak the
language and to know about the culture, literature, history in which
the language is a part of. The purpose of evaluation is the improvement
of the quality of our program.
II.
Intended uses, users, and evaluation questions
The intended users are faculty, students, and the Dean. The intended
use is to improve our program by redesigning our curriculum, changing
our teaching methodology, etc. The Dean may base any decisions
(monetary, related to new faculty and new classes, etc.) related to our
program based on the information produced by the evaluation. Students
may take any necessary steps (additional classes, more time studying
abroad, etc.) in case the student does not reach the intended level.
Prospective students may base their decision on the success of our
former students. The following are the questions for evaluation:
Q1: To what degree do our
students graduate with the basic knowledge we want them to have?
Q2: To what degree are we providing them with the necessary resources
for them to graduate with the basic knowledge we want them to have?
Q3: To what degree are our students able to compete with other students
coming out of similar programs?
III.
Methods, tools, and procedures
Q1: Structured interviews to be carried out in the student’s
junior year. The student’s advisor and at least one more
faculty member of UEFL program are present during the interview. The
interviews are designed to elicit the student’s attitudes and
self-evaluation of their own learning experience. The portfolio is
assessed in their senior year possibly during the capstone course. The
portfolio will include a variety or materials: (a) a self-reflection
written piece that the student will have to present to his/her advisor
and at least two more faculty members of UEFL program. This essay will
have to address the student’s learning experience at UE and
comments on the papers included in his/her portfolio (b) at least three
term papers dealing with literature, linguistics and
culture/history/current events (c) any other additional piece of
writing (term paper, self-reflection piece, essay, creative writing,
publication, etc.) that the student thinks it is important to include.
Q2: Analysis of assessment results. The pattern in students’ results will determine the outcomes that are receiving enough attention in our curriculum and the ones that need more support in terms of curriculum rearrangement.
Q3: Unstructured interviews with alumni and chairs of departments of other institutions our size.
IV.
Intended value added
I consider the value of this project has to do with the definition I
gave this assessment project itself: the ongoing process of reflecting
upon what we do in our program, based on systematic and objective
evidence, promoting internal (among faculty and students) and external
(administrators) communication, information and education; and judge
the effectiveness of our program to better determine its development
and improvement.
Program Evaluation with Respect to State-Required Outcome Performance for Certification
I.
Program description and overall purpose for evaluation
The Department of Foreign Languages, Literatures, and Cultures of
Central Michigan University offers a B.S. in Education for majors and
minors of French, German, and Spanish. All FL language majors and
minors on teaching degrees are presently required to pass a Simulated
Oral Proficiency Test at the Intermediate-High level of the ACTFL scale
before entering the FL methods course. This is an institution test,
created and assessed by the 24 members of the FL department, some of
whom have been formally, though minimally trained several years ago. It
has become increasingly important to evaluate the effectiveness of our
program’s speaking outcomes, especially since the State of
Michigan has changed the FL teacher certification policy, and will next
year require performance at the Advanced-Low (AL) level of the ACTFL
OPI, a level also required for NCATE certification
The purposes of the evaluation project are (a) to determine how well the FL Department is preparing its majors and minors in education to pass the new state certification criteria, and (b) to determine what changes are needed to help students achieve that level.
II.
Intended uses, users, and evaluation questions
Intended users are majors and minors in program and faculty of FL
department. To a lesser degree, they include the Department of
Education as well as high school programs and the future students these
people will be teaching. Intended uses center upon improving (a) the
certification percentage of graduating seniors, (b) the general FL
program, (c) student satisfaction with program, and (d) student
recruitment into the FL program. The following evaluation questions are
posed:
Q1: Can graduating FL education
majors and minors pass the OPI at the Advanced-Low level?
Q2: Where in the program are students getting the opportunity to
develop the competence needed to pass this test?
Q3: What program changes are needed to assure this level of proficiency
upon graduation?
III.
Methods, tools, and procedures:
Question 1: Can students pass
the OPI at the Advanced-Low level?
(a) Analysis and
examination of existing documents and reports. How
many students in the past have not passed parts of the
Department’s SOPI? How many tries? What parts of the test
have they passed and not passed? What were examiner comments (error
analysis)?
(b) ACTFL OPI.
Graduating seniors will take the OPI to determine their level. Results
will be compared to how they scored on the institutions’
SOPI.
Question
2: Where in the program are opportunities to develop the competence
needed to pass this test?
(a) Focus Groups (seniors)
Sample questions: What classes helped them? What were tasks in these
classes? Did they attend conversation hours, etc? What do they do
outside of class to increase proficiency? Have they studied abroad? How
many hours of contact do they have with TL each day? How did they
prepare for the SOPI? Suggestions for improvement?
(b) Questionnaire followed
by Meeting (faculty) to
determine how proficiency is addressed in various classes (syllabi,
materials, in and out of class tasks, methods of evaluation)
(c) SOPI at
200, 300 and 400 level for sample student groups.
(d) Samples of student
oral production in various
required courses at different levels evaluated by faculty.
Question
3: What changes are needed to insure this level of proficiency?
(a) Use of previously
discussed Senior Focus Group and Faculty Discussion group
(b) Survey (of alumni now
teaching) Possible
questions: What helped prepare you to teach in the TL and specifically
to speak proficiently? What would you suggest to improve the program?
(c) Analysis and
examination of existing documents and reports.
Examine syllabi to find out how course content and requirements relate
to proficiency standards
IV.
Intended value added
This will not only increase articulation between members of the faculty
regarding overall student learning outcomes and how to achieve them,
but also lead to a better articulated program as a whole. It will also
allow possible opportunities for faculty presentations and research
publications.
Student Satisfaction Survey to Measure Students’ Attitude towards Their Learning at Different Stages of Their German Studies
Annette Kym (akym@hunter.cuny.edu)Hunter College, CUNY, German Department
I. Program description and overall purpose for evaluation
Hunter College, with an
enrollment of approx. 20,000 students, is part of the City University
of New York. The German Department is a freestanding department with 5
FT faculty members, 1 TA and a number of adjuncts. The average
enrollment per semester is between 360 – 400 students, appr.
28 – 30% of those students are in upper-level courses taught
in the target language. Hunter College has a 4-semester language
requirement The German major requires 24 credits, the minor 12 credits.
We want to evaluate student satisfaction at various stages of our program. In this process, we want to find out what the value of learning outcomes is to students finishing the language requirement, and to students finishing upper-level courses. In addition, we want to find out how effective the department’s teaching approach is as judged by the students.
II. Intended uses, users, and evaluation questions
Intended
uses on the departmental level
By engaging in this evaluation, we want to find evidence on
students’ perceptions of the value of their language studies.
The information will help the department to take actions to improve the
program. It will also be used in discussions with the administration
about resources allocated to the department.
Intended
uses on the college level
We want to use the data in the broad discussion about the General
Education Requirement, which is currently under review. The
systematically collected data will produce evidence of the value of the
FL requirement as perceived by the students. Furthermore, the college
is undergoing accreditation by the Middle States Association and this
type of information can be used for that purpose.
III. Methods, tools, and procedures
We plan to collect the data through focus groups with students and student surveys. We will run a pilot project in the fall semester 2007, and conduct the focus groups and administer the finalized survey in spring 2008. The data will be analyzed over summer 2008 (Middle States Review will take place in Fall 2008).
IV. Intended value added
The German Department has
engaged in outcomes assessment for a number of years and we have data
about student learning outcomes. However, we have not collected data
from the students’ point of view and measured
students’ perception about their learning outcomes
systematically. Getting data on their perceptions will fill this gap,
and in conjunction with other indicators, will help us to improve the
program and assist us in long-term program planning.
Evaluation of a Spanish Pilot Project in the Center for Languages and Cultures
Beth Landers (landersb@umsl.edu)Director of the Center for Languages and Cultures,
University of Missouri-St. Louis
I.
Program description and overall purpose for evaluation
The Center for Languages and Cultures is located within the department
of Anthropology and Languages at the University of Missouri-St. Louis.
UMSL is an urban, public research university with high numbers of
non-traditional students who commute to campus. The Center is
responsible for elementary and intermediate-level language courses in
nine languages (Spanish, French, German, Japanese, Chinese, Modern
Greek, Arabic, Latin and Ancient Greek) as well as ESL. Approximately
40% of all UMSL undergraduates have a three-semester foreign language
requirement (all students who earn BA’s, as well as a few
select other degrees). The Center serves these students, and also
endeavors to retain students beyond the three-semester required
sequence, preparing them to major or minor in language.
The specific component of the program to be evaluated is a Spanish Pilot Project initiated and funded by the College of Arts and Sciences as part of an initiative to improve all required classes in Math, Foreign Languages, and English. This particular project focused on ways to improve the oral and written proficiency of students enrolled in the first three semesters of language courses (the required sequence). At its inception, no provision was made to evaluate the pilot project but it was intended to inform curricular changes in all lower-division classes across language sections.
II.
Intended uses, users, and evaluation questions
This evaluation will identify best practices from the project that can
be generalized and applied to all sections of elementary French, German
and Spanish. The users will thus be the Director of the Center and the
thirteen instructors in these sections. Specific questions to be asked
are: What did the project do that was different from the past? What was
innovative? What was traditional? What worked well and why? What did
the teachers like? What did the students like? How did the Pilot
Project affect the oral and written proficiency levels of students? How
did the Pilot Project affect student success rates? Did the Pilot
Project affect student motivation?
III.
Methods, tools, and procedures
In order to answer these questions we will use a variety of methods: 1)
a questionnaire distributed to instructors 2) a focus group of the
three instructors at the end of the project 3) analysis of video tapes
of particularly innovative class sessions 4) a student questionnaire
distributed to 3rd semester Spanish students in fall 2007, 5)
comparison of OPI ratings of the control group and the pilot project
group in their third semester, 6) comparison of compositions on final
exams with compositions from previous years, 7) analysis of course
materials developed over three semesters, 8) analysis of student course
evaluations, 9) analysis of student attendance/absence patterns 10) a
report from the office of Institutional Research about the pass/fail
rate of students over the three semesters of the pilot project, 11) a
report from the OIR on how many students from this group enrolled in
Spanish 4 in spring 2008.
IV.
Intended value added
This evaluation will allow us to target the best practices developed in
the Spanish Pilot Project and incorporate them in all basic language
classes in order to refresh teaching practices, bring a measure of
standardization to basic language instruction, and improve student
learning outcomes. We hope that doing so will increase the number of
students who continue their language study beyond the third semester.
Ultimately, we hope to encourage further investment in the Center for
Languages and Cultures by assuring the College that its initial
investment in the Spanish Pilot Project has been used in constructive
ways.
Evaluating the Foreign Language Requirement at Duke University
I.
Program description and overall purpose for evaluation
In the year 2000, Duke University instituted a new curriculum for
undergraduates of Trinity College (the undergraduate college of the
School of Arts and Sciences), which includes a foreign language
requirement. Whereas the last several years have been devoted to
addressing issues of implementation, the Dean of the College is now in
the process of evaluating the curriculum. Most recently, a committee
consisting of several language program directors and the director of
the assessment office has been asked to determine a means of
documenting student learning outcomes, for the purpose of understanding
the extent to which students are achieving the goals articulated by the
foreign language requirement.
II.
Intended uses, users, and evaluation questions
The primary uses of the evaluation project are to demonstrate the merit
and value of the foreign language requirement by documenting student
learning outcomes, to chart student progress at various stages of their
foreign language learning, and to improve foreign language programs and
curricula. The information obtained from the evaluation project will
primarily be used by university administrators to document what
students are learning as a result of the foreign language requirement,
and possibly to more clearly identify and understand how individual
language programs prepare students to meet the foreign language
requirement goals established by the university. The information
gathered will be used by individual language program faculty to assess
the extent to which their own programs and curricula are preparing
students to meet these goals, for the purposes of engendering
programmatic and curricular improvement. In particular, we would like
to investigate the following questions: 1) What levels of language
proficiency are students achieving in speaking, reading, and writing?
2) What cultural knowledge, understandings, and perspectives are
students gaining at each level of the curriculum? 3) To what degree do
factors such as language program curricula, study abroad, heritage,
previous language learning experiences play a role in
students’ learning outcomes? 4) What are student perceptions
of the university foreign language requirement?
III.
Methods, tools, and procedures
We will be using multiple methods to collect the information we need,
including SOPI or COPI, course embedded assessments, student
portfolios, online questionnaires, individual interviews and focus
groups. We will most likely collect this information in stages: using
the fall semester of 2007 for planning and piloting, the spring
semester of 2008 for initial data collection, and the following
academic year for analysis and additional data collection, as informed
by decisions based on collected data.
IV.
Intended value added
We imagine that the very process of engaging in the evaluation project
will help individual language programs address issues of curricular
goals and coherence, faculty development, and articulation with other
programs and departments across the university. We expect that the
project will help communicate more precisely to the outside world
(administrators, faculty, students, parents, communities) just what it
is our foreign language programs do and what students learn. This may
enhance the profile of foreign language programs within the university
and beyond as an integral part of the humanities in a liberal arts
education. And finally, we hope that this project will contribute to
the growing research in the field of foreign language education and
program evaluation.
Assessment of Student Learning Outcomes for the Spanish Major
Judith E. Liskin-Gasparro (judith-liskin-gasparro@uiowa.edu)
University of Iowa, Department of Spanish and Portuguese
I.
Program description and overall purpose for evaluation
The undergraduate major in Spanish consists of 12 upper-division
courses: 1 course in each of 4 core areas (Peninsular literature,
Spanish American literature, Culture, and Linguistics) and 8 elective
courses. The purpose of evaluation is to provide evidence that will
inform faculty discussion about the effectiveness of the curriculum
and, thus, serve as the basis for making changes to enhance and improve
student learning. The immediate impetus for the evaluation is the
university-wide mandate to formulate an outcomes assessment plan for
the major for the upcoming reaccreditation cycle.
II.
Intended uses, users, and evaluation questions
Intended uses
(a) Make changes in the curriculum (types of courses, content of
existing courses)
(b) Increase number of/access to curricular components most highly
valued by students
(c) Review (and possibly adjust) the departmental hiring plan in light
of the evaluation
(d) Identify additional areas that would benefit from evaluative
attention
(e) Provide evidence of program quality and departmental commitment to
improvement of student learning for internal and external proposals
(e.g., faculty lines, funding)
Intended users: Faculty, students, administrators
Evaluation
questions
Q1. How effective is our curriculum in teaching speaking skills?
Q2. How effective is our curriculum in teaching high-level writing
skills?
Q3. Which components of the program do students value most highly as
contributing to their learning?
III.
Methods, tools, and procedures
(a) Speaking
(Q1): Locally designed
interview procedure that focuses on Intermediate Mid?Advanced. The
outcomes assessment (OA) committee will produce an interview protocol
and train a small number of faculty in the procedure. Speech samples
will be designated as Meets Expectations (Intermediate High), Exceeds
Expectations (Advanced Low or higher), or Does Not Meet Expectations
(Intermediate Mid or lower).
(b) Content-based writing
(Q2): Work samples (e.g.,
research papers) embedded in senior-level courses will be assessed for
the ability of the writers to produce in Spanish (i) coherent analyses
and interpretations of Spanish/Latin American literary texts or
cultural phenomena, (ii) coherent summaries and evaluations of
linguistic phenomena and construction of linguistic arguments, and
(iii) coherent analyses of Spanish/Latin American films. Students will
create electronic portfolios that include 2 papers that they believe
represent their best work from 1 or more of their 3 highest-level
courses. A sample of these papers will be assessed by the OA committee
(Meets Expectations, Exceeds Expectations, and Does Not Meet
Expectations) according to a rubric designed by the committee for this
purpose.
(c) Student perceptions
of value of program (Q3):
Students will write for their portfolios a reflection essay, in which
they will be asked to reflect on the aspects of the program that
contributed most significantly to growth in their (i)
intellectual/academic knowledge and analytical ability, (ii) linguistic
skills, and (iii) understanding of diverse cultural perspectives. Focus
groups will be held with a sample of graduating majors to address the
same topics. Reflection essays and focus group transcripts will be
analyzed for key themes related to points (i), (ii), and (iii).
IV.
Intended value added
(a) Provide an evidentiary basis for understanding the program and how
it functions (evaluation) and to produce/implement a plan for change
(actions) to increase its effectiveness to enhance student learning.
(b) Communicate the value of our program to administrators, students,
external agencies, etc.
(c) Provide a site for research on teaching and learning in foreign
language programs.
Foreign Language Enrollment Growth Initiatives at Cal State L.A.
Sachiko Matsunaga (smatsun@calstatela.edu)
Chair
of Department of Modern Languages and Literatures,
California State University, Los Angeles
I.
Program description and overall purpose for evaluation
Our Foreign Language Program, which belongs to the College of Arts and
Letters, consists of MA programs in French and Spanish, BA programs in
Chinese, French, Japanese, and Spanish, and Subject Matter Preparation
programs in French, Japanese, and Spanish, as well as elementary level
courses in Arabic, Italian, Korean, and Portuguese. The main components
that constitute our Foreign Language Program are: courses, teaching,
and students in pedagogical arrangement; staffing and finance (for
instructors, space and equipment) in institutional arrangement.
II.
Intended uses, users, and evaluation questions
Intended users are:
primarily full-time faculty (CHIN; FREN; JAPN; SPAN), Department Chair,
and Dean; secondarily part-time faculty, students, other departments,
senior administrators, local schools, and communities.
Intended
uses are (a) faculty act on
the evaluation results (e.g., modify courses/programs; identify new
practical opportunities); (b) the Department Chair seeks support from
the Dean, who decides whether to advocate our FL Program to the senior
administrators to fund new initiatives; (c) senior administrators
decide whether to increase funding for our growth initiatives; and (d)
local schools and communities decide whether to express support for
potential changes to be made based on the evaluation findings and our
plan.
The first question of evaluation is: What are other CSU campuses doing in serving and recruiting FL students? We ask this question in order to know why our overall enrollment has declined (our SFR is lower than the average CSU SFR). Possible reasons could be environmental, institutional, and/or programmatic. The findings will help us offer explanations to the administrators. The second question of evaluation is: In what areas can we expect enrollment growth? We ask this question in order to know what actions we need to take in order to increase FL enrollments on our campus. The findings will be used to request support from the administrators.
III.
Methods, tools, and procedures
Methods that we will employ are: (a) Internet search for factual
information about other CSU campuses; (b) questionnaire distributed to
all CSU FL Department Chairs electronically to find out what they are
doing in serving and recruiting students; (c) faculty meeting with the
Department Chair as a facilitator to come up with areas for growth; (d)
focus group discussions to investigate whether the areas for growth
identified by faculty match the needs of learners (current majors and
non-majors as well as alumni), local schools, and communities.
Responses on the CSU survey will be collected electronically, and
(follow-up) phone interviews will be conducted if necessary. We will
use Strengths Weaknesses Opportunities and Threats analysis to analyze
all data, and present the results to the Dean and senior administrators
in order to convincingly communicate to them the importance of the role
that we play within the University and the necessity of funding for the
identified growth (with supporting evidence gathered from all
stakeholders) to happen.
IV.
Intended value added
If we successfully gain support from the administrators with these
initiatives, we will be able to not only strengthen our existing
programs for our majors with new hires, more space, and better
equipment, but also further diversify our course offerings for
non-majors. In addition, the role that we play within the University
will be raised, possibly to the extent that the College or University
might consider implementing a foreign language requirement.
Furthermore, it will prepare the Department well for the program review
that we need to start conducting 18 months from now.
Evaluating the Modern Languages and Linguistics Department at UMBC
I.
Program description and overall purpose for evaluation
The Modern Languages and Linguistics (MLL) department at the University
Maryland Baltimore County (UMBC) emphasizes communication across
boundaries. The MLL program has been praised for its innovative and
interdisciplinary nature, but since 1985, the program has expanded in
both scope and amount of students. Recently, the faculty started
re-examining the curriculum for articulation purposes. Still, there is
a need to frame the MLL curriculum within the university mission
statement, and to find means of documenting learning outcomes that fit
with the department and the university goals.
II.
Intended uses, users, and evaluation questions
Intended uses
(a) To examine the extent to which MLL department’s (and each
language area) mission statement and goals fit with the university
mission statement and goals.
(b) To establish a common understanding in the MLL department (and in
each language area) of “knowledge,”
“skills,” and “dispositions.”
(c) To establish expected student learning outcomes in the MLL
department (and in each language area).
Intended
users
The intended users are faculty, current and prospective students, high
schools, and administration.
Evaluation
questions
(a) How do the MLL department’s objectives fit with the
University mission?
(b) How do the Spanish area objectives fit with the MLL department
mission?
(c) What is the common understanding of
“knowledge,” “skills,” and
“dispositions” that MLL students need?
(d) How does the curricula (and specific classes, curricula, syllabi)
in the Spanish area provide students with the knowledge and skills
needed to achieve the program objectives?
(e) The fifth question is divided into 2 questions:
1. What are the MLL department student outcomes regarding writing,
speaking and culture?
2. How does the Spanish area determine whether Spanish area students
have achieved expected program outcomes (writing, speaking and culture)?
(f) How well does the program communicate program goals to the administration, other departments at UMBC and students (current and prospective)?
III.
Methods, tools, and procedures
The department will used different methods/tools to collect information
needed such as (a) document analysis to understand alignment of mission
statements, (b) discussions with faculty to define relevant concepts,
(c) questionnaire to alumni to gain information regarding the validity
of the program mission statement and perceived program value, (d) foci
group with students to elicit information regarding program
perceptions, program components (speaking, writing, culture, etc.), and
relationship with faculty/program, (e) final papers and class
discussions to observe accomplishment of culture learner outcomes, and
(f) short questionnaires to administration and other departments to
gain information regarding professional relationship with other
departments, and administration.
IV.
Intended value added
The intended value added of the evaluation is to increase awareness of
how the MLL department addresses issues of curricular goals and
coherence, and articulation with other programs and departments across
the university. The evaluation will also emphasize the MLL program
identity to outside stakeholders (administration; alumni; prospective
students, etc.), and it will highlight its specific contributions to
the students’ educational experience within and beyond UMBC.
Tailoring Language Majors’ Year Abroad Experience to Improve Outcomes
Violeta Ramsay
(vramsay@linfield.edu)
Chris Keaveney (ckeaven@linfield.edu)
Co-Chairs
of Department of Modern Languages,
Linfield College
I. Program description and overall purpose for evaluation
The Department of Modern Languages at Linfield College offers minors and majors in French, German, Japanese, and Spanish (as well as first year courses in Chinese and Latin). The department has ten full-time faculty and 1-3 adjunct instructors per year. Language requirement for a B.A. degree is one year. The purpose of our evaluation is to improve the level of achievement of all majors.
II. Intended uses, users, and evaluation questions
The intended uses of evaluation are (a) to make sure all language majors have the same opportunities to reach the goals of the major; and (b) to improve students’ year-abroad experience. Our evaluation question we have posed is: What aspects of student’s preparation, the year abroad experience or the design of the capstone seminar cause wide disparities in levels of achievement among language majors?
III. Methods, tools, and procedures
These
are the steps we take in the evaluation process:
(a) Design a manual for majors that contains information on all stages
of the major program
(b) Select programs that best match our program and goals; establish
communication with those programs, and ask students to select among
them.
(c) Collect data from students to establish a basis for the observation
of development: This includes samples of written work and an oral
proficiency interview (OPI) before students leave.
(d) Increase contact with and supervision of students’ work
while they’re abroad, creating a one-credit course to make it
possible. The course calls for the following:
• Students samples of
papers and two essays
• A second OPI by phone
• Initial work on senior project: choosing topic, making
outline, collecting data
• A third OPI upon return to home campus
(e) Make changes in capstone seminar to allow for more contact with advisors as well as more time, and guidance during the writing, revisions and presentation of senior project.
The
following timeline guides the procedures
(a) Majors manual is given to students when they declare a major and
reviewed before departure
(b) Pre-departure data collection takes place during
students’ 4th semester
(c) Data collection from abroad takes place during 5th and 6th semester
(d) Guidance in capstone seminar takes place during students’
7th semester
IV. Intended value added
Improvement in the
communication with and guidance for students during their year abroad
(most critical year for the major). The junior year experience can be
tailored to better suit the rigor and quality of our program at home.
Preparation for
Department Evaluation:
Building Faculty Capacity in Self-Evaluation
Montserrat Reguant (mreguant@msmc.la.edu)
Mount Saint Mary’s College, Los Angeles, Department of Language and Culture
I.
Brief description of the program and overall purpose for evaluation
To prepare the Department of Language and Culture (LC) at Mount Saint
Mary’s College to respond to the cyclic evaluation required
by the institution, and at the same time to build faculty capacity in
self-evaluation in order to improve the effectiveness of the whole
department.
II.
Intended uses, users, and evaluation questions
The intended use of the evaluation activity is to improve the
department as a whole over time, as preparation for the evaluation of
department in spring of 2008. The intended users are faculty members of
the Language and Culture Department and Academic
Vice-President-Provost.
Evaluation question: How can the Department of LC self-improve over time, in preparation for the departmental evaluation in 2008 and build faculty capacity in self-evaluation?
III.
Methods, tools, and procedures
We are going to utilize self-assessment, using given indicators by the
Curriculum Committee on department goals, curriculum, faculty, and
student assessment, and institution. Surveys have already been
prepared. They will be distributed to the faculty of the LC department,
filled out, and collected during the first department meeting of
2007-2008 academic year, August 27, 2007. Data will be analyzed in the
following three weeks and action for improvement will start before the
end of the fall semester. The following is a selection of indicators
included in the surveys:
(a) Department goals: Do they address the needs of the students, the College Mission or strategic planning, long term planning, student assessment, and student retention?
(b) The
Curriculum:
1. How does it support the department goals?
2. How does it meet the needs of majors and minors?
3. How does it compare to that of a similar institution?
(c) Faculty: Commitment to teaching, strong collegial environment, innovation in pedagogy, quality in course content, indicators of student success, on-going effort to strengthen the department, assessment of the department’s curriculum, integration of full-time and adjunct faculty, assessment of the department’s curriculum, assessment of the quality of the major
(d) Student Assessment:
1. The department objectives and assessment methods are aligned
2. Methods of assessment across sections
3. Embedded techniques of assessment across sections and/or courses
4. Student feedback is used in regular basis to assess the efficacy of
student assessment in
the department A variety of methods are used as part of student
assessment
5. Use of at least one type of student-driven student assessment method
(e.g., electronic portfolio, senior capstone project, etc.)
(e) The Institution: Library offerings and services, tech resources and support, student support services, faculty support services, professional development and other enriching opportunities, equipment and facilities, and balance of full-time and adjunct faculty.
IV.
Intended value added
This project will expand the culture of evaluation preparation for
departmental self-evaluation as an improvement tool. This
self-reflection for departmental improvement will ensure we are well
prepared for the external evaluation and at the same time we hope to be
able to build faculty capacity in self- evaluation to enhance student
learning.
Development and Evaluation of Arabic Teacher Development Program
Martha Schulte-Nafeh (marthas@u.arizona.edu)
University
of Arizona,
Center of Middle Eastern Studies and Department of Near Eastern Studies
I.
Program description and overall purpose for evaluation
The Arabic Graduate Teaching Assistants (GTAs) have all attended the
same teacher training workshop in which a particular
“vision” of best language teaching practices was
presented and discussed. The purpose of the professional development
program is to guide the GTAs to a fuller understanding of those
practices and to increase their ability to implement them. The GTAs
will engage in 6 teacher development activities: (a) participation and
reflection on lesson planning meetings, (b) reflection on their own
planning and teaching, (c) observation of colleagues, (d) videotaping
and reflective viewing of their own teaching, (e) seeking formative
student feedback on their teaching, and (f) being observed by the
coordinator. The teacher development program is being developed
“evaluatively”; That is, by considering evaluation
questions throughout the planning process and by including an explicit
plan for formative and summative evaluation of the program. The
evaluation component of the plan is described below:
II. Intended uses, users, and evaluation questions
Intended
uses
(a) To determine and document the success of the program in developing
GTAs understanding of and ability to deliver language instruction that
is in line with the vision.
(b) To identify areas of weakness/possible improvement of the plan from
all stakeholders’ perspectives in order to further develop
the plan.
(c) To inform the development of the vision itself as well as the way
that it is taught to new GTAs.
Intended
users
The Arabic GTAs, The ME language coordinator, the director of the
Center for Middle Eastern Studies and the head of the Near Eastern
Studies Department
Evaluation
questions
(a) Does the development plan lead to increased teaching of Arabic
based on the identified good teaching practices?
(b) Does the development plan lead to increased sense of agency?
(c) Does the development plan lead to an increase in reflective
language teaching?
(d) Which of the activities in the plan are most useful and why?
(e) Is the development plan meeting the users (GTAs) needs?
III. Methods, tools, and procedures
A variety of data sources will be used to answer the questions including: qualitative analysis of GTAs written reflection on their lesson planning and teaching; questionnaires for GTAs in the middle of the training period and at the end; faculty classroom observations of GTAs teaching; student questionnaires
IV. Intended value added:
The evaluation process will
provide information from students, GTAs, and other faculty which can
contribute to the articulation and development of a program wide
understanding of good teaching practices.
USCB Spanish Evaluation: “From Growing Pains to Student Gains”
Babet Villena-Alvarez (jivillen@gwm.sc.edu)
University of South Carolina Beaufort, Foreign Language
I.
Program description and overall purpose for evaluation
(a) Student Enrollment (approximate): Certificate (8 courses): 15
students; Minors (9 courses): 8 students; Majors (10 courses for
Heritage track or 16 for General major): 13 students
(b) Institutional Background: USCB—newest baccalaureate
degree-granting institution of state; 1,200 students in
state’s fastest growing region.
(c) Program Description: Spanish degree is 9th degree approved by the
South Carolina State Commission on Higher Education (SC-CHE) in January
2006.
The overall purpose is “to evaluate how well the Spanish curriculum at USCB prepares its students in the major, minor and certificate programs.” This evaluation is summative (to judge and value) and formative (to develop and improve).
II.
Intended uses, users, and evaluation questions
Intended uses
(a) Improve Program (Quality of Spanish curriculum)
(b) Conduct Outcomes Assessment and Institutional Effectiveness
(Quality of faculty)
(c) Promote Student Advancement, Retention, and Recruitment (Quality of
students)
(d) Support other university programs (Quality of university
curriculum).
Intended
users
Spanish Faculty (Full-time & Part -time), Office of
Institutional Effectiveness, Office of Academic Affairs, Spanish
students, community (local businesses & offices).
Evaluation
questions
Overall question: How well does the Spanish curriculum prepare its
students in the major, minor and certificate programs?
(a) What do major/minor/certificate students need to know to
communicate effectively in Spanish in their career choice?
(b) With respect to cultural competency, how capable are the students
in distinguishing Spanish/Iberian cultures from the Hispanic/Latin
American cultures?
(c) With respect to the major/minor/certificate in Spanish, what are
the students’ goals and motivations?
(d) To what extent do students in the major/minor/certificate programs
participate and enhance their cultural knowledge through activities and
resources outside class (i.e., movies/audiovisual materials in the
target language, club activities, international festivals and other
events, and volunteer work where the target language is used)?
III.
Methods, tools, and procedures
Method
Survey – electronic and/or paper (conduct a pre-program and a
post-program survey); exit interview with the Program Director or a
Full-time Faculty; tests – in-course; testimonials
– at the end of the exit interview; Institutional
Effectiveness and Outcomes Assessment.
Tools
Survey instrument – two questionnaires; exit interview; list
of cultural content items relating to the specific items to be
evaluated that need to be embedded and incorporated in specific course
syllabi and tests; form to be completed for testimonials; Institutional
Effectiveness and Outcomes Assessment document for the Office of
Institutional Effectiveness (For accreditation purposes).
IV.
Intended value added
The main directive from the USCB administration for the coming academic
year is to grow the Spanish Program. In the process, an effective
evaluation plan needs to be embedded in the growth process to ensure
on-going curricular improvement, to put in place faculty
accountability, and to substantiate student language proficiency.
Evaluation of the
Arabic and Persian Summer Language Institute,
Georgetown University
Kassem Wahba (kmw@georgetown.edu)
Georgetown University, Department of Arabic and Islamic Studies
I. Program description and overall purpose for evaluation
In collaboration with the
Department of Arabic and Islamic Studies, the Summer School in the
Department of Continuing Education offers an Arabic language program
within the Arabic & Persian Summer Institute. The Arabic
program is an undergraduate language program of intensive and
non-intensive courses which last for ten weeks. The courses offered
include basic, intermediate and advanced level Modern Standard Arabic
and elementary classes in the Iraqi, Levantine, and Egyptian Arabic
dialect. The number of students enrolled for Arabic and Persian is
approximately 60 and 13, respectively. The students’
motivations for taking Arabic include wanting to work for the
government, wanting to specialize in the Middle East, or fulfilling
education requirements.
The overall purposes of evaluation are (a) to offer the administration information regarding the program’s curriculum structure and its impacts, and (b) to ascertain the effectiveness of the curriculum.
II. Intended uses, users, and evaluation questions
Intended
uses
(a) To find out how well the current Arabic language curriculum meets
students’ needs
(b) To identify what is missing in the curriculum and to propose a plan
to implement necessary changes for improving the curriculum
Intended
users
Various administrators responsible for the program at the level of the
university, within the department of Arabic and Islamic Studies and the
Summer School of Continuing Education.
Evaluation
questions
(a) What are the learners’ needs in studying Arabic?
(b) What are the current student learning outcomes?
(c) Is there a match between the outcomes and the learners’
needs?
(d) What changes are needed in order to better respond to
learners’ needs?
III. Methods, tools, and procedures
Needs analysis will be carried out to gather information with regard to administrations, faculty, and learners’ perceptions of the students’ learning needs. The tools to be used are: interviews and a questionnaire.
IV. Intended value added
(a) To increase awareness among
the administration and the faculty of the potential need to modify the
Arabic language curriculum to meet learners’ needs.
(b) To understand and improve the current program practice.
(c) To focus on the effectiveness of teaching and learning in terms of
learning outcomes.
Identifying
a Nationally Accepted Tool to Assess Student Learning Outcomes in the
LCTL’s at the University of Florida
Ann Wehmeyer (wehmeyer@aall.ufl.edu)
I. Program description and overall purpose for evaluation
The goal of the program is
twofold. First, we wish to determine whether student outcomes in our
foreign language programs are comparable to those in external foreign
language programs. Second, we want to ensure that training capacities
for nationally accepted proficiency testing are incorporated into all
of our language programs (Akan, Amharic, Arabic, Chinese, Hebrew,
Hindi, Japanese, Korean, Swahili, Vietnamese, Wolof, Xhosa, Yoruba).
II. Intended uses, users,
and evaluation questions
The users of the evaluation program are department faculty, undergraduate and graduate students of language courses, area studies center directors, CIBER director, Dean of International Studies, and accreditation agencies. For faculty, the evaluation will serve to clarify our respective statements on student learning outcomes, and to identify a nationally recognized assessment tool which can be effectively applied to assess one or more of these outcomes. It will also enable us to incorporate into our language program features in common with external language programs. For students, the evaluation will serve to provide them with an opportunity to obtain nationally recognized language credentials. For administrators in related programs and centers, the evaluation will supply data that can be used to substantiate claims about the quality of our foreign language programs for purposes of grant application and reporting. For accreditation, the evaluation will serve to put in place an ongoing system of monitoring and assessment.
The overarching question underlying this evaluation program is to determine whether student outcomes in our language programs are comparable to those in external language programs. We will need to take the following steps in order to address this larger question: (a) What are the student learning outcomes in each of our respective language areas?, (b) What is the most commonly accepted assessment tool in each of the language areas?, (c) Is it feasible to apply this assessment tool to all students at selected levels of language? The answer to question (c) will require an investigation of fit to student learning outcomes, cost of materials, and cost of human resources investment in training and rating.
III. Methods, tools, and procedures
While languages that are part of a major have explicit student learning outcomes, those that are not yet part of a major lack formal statements in this regard. The language coordinator of each language unit will work with faculty in the unit to articulate, or revise, a formal statement of student learning outcomes. Secondly, each unit will determine which assessment tool among those currently available (e.g., for oral proficiency, ACTFL’s OPI, SOPI, COPI; for reading and writing proficiency, Oregon’s STAMP, etc.) is most commonly applied in their language area. In conjunction with the chair, each language unit will determine the costs of materials, and costs of human resources investment. Purchase of materials and training will commence along a timeline of feasibility. The selected assessment tool will be applied toward the end of the second year of language study, and also toward the end of the final semester of language study in those languages that are part of a major program. Results will be reported to all users.
IV. Intended value added
As faculty become knowledgeable
about national assessment measures, their ability to incorporate skills
targeted by nationally recognized tests into their respective language
curricula will be enhanced, thereby bringing our programs in the LCTLs
more in line with the goals of programs in the more commonly taught
languages.
Use of Evaluation for Developing Russian Summer Abroad Program
Valentina Zaitseva, (vaz2@u.washington.edu)
Lecturer,
University of Washington,
Department of Slavic Languages and Literatures
I. Program description and overall purpose for evaluation
The UW Department of Slavic Languages and Literatures has made a decision to offer a Russian summer abroad program in Sochi and to model it after the existing Czech summer program in Prague (set as Exploration Seminar, 3 weeks, 6 credits). The overall purpose of the evaluation project is to evaluate and inform the development of Russian Summer Program Abroad (Sochi).
II. Intended uses, users, and evaluation questions
The intended uses of this project are (a) to establish how the Russian Summer Program would contribute to the goals and missions of the Department; and (b) to evaluate how the Summer Program in Sochi would meet the learners’ needs and satisfy the requirements of Social Science Research Council (SSRC) funding for Russian Summer programs.
The intended users of the evaluation are chair, administrator and senior faculty, Russian summer program faculty; SSRC, intermediate and advanced UW students, graduate students, and faculty and administrators in Sochi University.
Evaluation
questions
(f) What are pros and cons for setting a 4-week intensive language
program (45 contact hours equivalent of one quarter)? Which models for
summer program would better meet the students’ needs,
departmental standards, and SSRC requirements?
(g) What kind of curriculum would meet the set goals? How can we build
in special projects in under-/graduate research using local Sochi
opportunities?
(h) How can local Sochi opportunities be used in task-based language
teaching or special socio-cultural projects curriculum?
(i) Which components of the Czech summer program in Prague and
recruitment strategies are transferable to Russian program in Sochi?
(j) How would Russian Summer courses fit curricularly into the overall
Slavic Department program? What implications for regular curriculum
would Summer program entail?
(k) What outcome assessment should be implemented in the end of the
Russian Summer program?
III. Methods, tools, and procedures
We are going to use learner needs analysis (through survey questionnaires and analysis of program requirements). We will collect pertinent data and perform a document analysis (UW program Abroad and other FL institutions websites, information about the Sochi University). Individual interviews, meetings and focus group discussions will help us to build consensus on length, intensity and curriculum of Russian summer program in Sochi. Informal preliminary meetings with faculty in Sochi in August, 2007 would outline the specific evaluative points to be incorporated into the next cycle of evaluative and program-building activities.
IV. Intended value added
While the development of the Russian Summer Program Abroad is in its very initial stage, the evaluative approach to program building helps to address a broad range of topics and issues in a practical and specific way, taking into consideration needs and improvement of the entire Slavic Department program.
Evaluating the Need for Formal Graduate L2 Teacher Training
Alessandro Zannirato (alessandro1@jhu.edu)
Johns
Hopkins University,
Department of German and Romance Languages and Literatures
I.
Brief description of the program and overall purpose for evaluation
The Italian section offers both majors and minors in Italian studies
and has a vibrant PhD program in Italian literature. Most of the
doctoral candidates are required to start teaching undergraduate
language courses right after being admitted into the PhD program. Many
of them have limited training in L2 teaching. Presently, the only
training that the GTAs receive takes place through weekly meetings with
the Program Directors or coordinators and through a reflection on
classroom observations. It may be wondered if this is sufficient, or
whether some sort of more formalized training may be advisable.
II.
Intended uses, users, and evaluation questions
Intended uses and users
Through this evaluation, it is endeavored to: (a) evaluate the
usefulness of GTA formal training in L2 language teaching in order to
improve the quality of the undergraduate program; (b) reflect on the
graduate curriculum and weigh the possibility to include formal
training in L2 language teaching; and (c) verify if the investment
needed to introduce formal L2 language teacher training would be
justifiable. The primary intended users are: Language Program
Directors, Section Heads and the Department Chair. Secondary intended
users would include the Dean’s Office and graduate students.
Evaluation
questions
The evaluation questions that were asked are: (a) Is it possible to
improve the quality of our undergraduate courses through structured GTA
training in L2 teaching? (b) Can we make the GTAs’ academic
experience a better one? (c) Could we provide our graduate students
with skills and formal training to become more competitive in the job
market? (d) Is our department following nationally-recommended
practices with regard to graduate student teacher training?
III.
Methods, tools, and procedures
Document analysis, focus groups, meetings and questionnaires will be
used. We will create an online questionnaire using Surveymonkey
software, and review the journal articles from the Chronicle of Higher
Education. The following are the steps we are going to take.
(a) Program directors could
review last year’s teaching observation reports across the
different languages, identify the repeating themes in order to get an
idea about the quality of GTA teaching and formulate recommendations.
(b) A focus group with one GTA from each language could be organized in
order to prepare a questionnaire to be distributed to all GTA, with the
objective to verify whether they are satisfied with their teaching
abilities. GTA responses may then be triangulated with program
directors’ recommendations.
(c) Administer a survey to literature faculty members to verify their
perceptions of L2 language teacher training and its relevance in the
graduate curriculum.
(d) Review recent job advertisements to ascertain whether formal
training in L2 language teaching is considered as a necessary or
desirable qualification by search committees.
(e) Identify a representative sample of peer institutions and review
what the role of L2 teacher training is in their curricula. Review FL
professional organizations’ recommendations (e.g. MLA/ACTFL
etc.).
IV.
Intended value added
This evaluation will hopefully contribute to a better understanding of
the role applied linguistics can play in today’s foreign
language departments.