resources

Using Surveys in Language Program Evaluation

I. How-to Books

II. Surveys in L2 Program Evaluation, Curriculum Development, & Research

III. Online Advice on Designing Surveys for Evaluation

IV. Comparison of Some Online Survey Software and Features

V. Research on the Use of Online Surveys

I. How-to Books

GENERAL

Alreck, P. L., & Settle, R. B. (1994). The Survey Research Handbook (2^nd ed.). New York : McGraw-Hill.

Dillman, D. A. (2000). Mail and Internet Surveys: The Tailored Design Method. New York: Wiley.

Fink, A., & Kosecoff, J. (1998). How to Conduct Surveys: A Step-by-Step Guide (2nd ed.). Thousand Oaks , CA : Sage Publications.

Fowler, F. J. (2003). Survey research methods (3^rd ed.). Thousand Oaks, Calif. : Sage Publications.

Fowler, F. J. (1995). Improving Survey Questions: Design and Evaluation. Thousand Oaks, Calif. : Sage Publications.

Henerson, M. E., Morris, L. L., & Fitz-Gibbon, C. T. (1987).How to Measure Attitudes (2^nd ed.). Beverly Hills, Calif.: Sage Publications.

Rea, L. M., & Parker, R. A. (2005). Designing and conducting survey research: A comprehensive guide (3^rd ed.). San Francisco : Jossey-Bass.

Schuman, H., & Presser, S. (1981). Questions and answers in attitude surveys: experiments on question form, wording, and context. New York : Academic Press.

LANGUAGE

Brown, J. D. (2001). Using surveys in language programs. Cambridge : Cambridge University Press.

Davis, J. McE. (2011). Using surveys for understanding and improving foreign language programs. (NetWork #61) [PDF document]. Honolulu: University of Hawai•i, National Foreign Language Resource Center. doi: http://hdl.handle.net/10125/14549

Dörnyei, Z. (2003). Questionnaires in second language research: Construction, administration, and processing. Mahwah , New Jersey : Lawrence Erlbaum Associates, Publishers.

II. Surveys in L2 Program Evaluation, Curriculum Development, & Research

Chaudron, C., Doughty, C., Kim, Y., Kong, D., Lee, J., Lee, Y., et al. (2005). A Task-based needs analysis of a tertiary Korean as a foreign language program. In M. H. Long (Ed.), Second Language Needs Analysis (pp. 105-124). Cambridge : Cambridge University Press.

Dörnyei, Z., & Schmidt, R. (Eds.) (2001). Motivation and second language acquisition (Technical Report No. 23). Honolulu : University of Hawai ‘i, Second Language Teaching and Curriculum Center .

Iwai, T., K. Kondo, D. J. Lim, G. E. Ray, H. Shimizu, & J.D. Brown. (1999). Japanese Language Needs Analysis 1998-1999. Honolulu: Second Language Teaching & Curriculum Center, University of Hawai ‘i. http://hdl.handle.net/10125/8950

Jasso-Aguilar, R. (2005). Source, methods, and triangulation in needs analysis: A critical perspective in a case study of Waikiki hotel maids. In M. H. Long (Ed.), Second Language Needs Analysis (pp. 127-158). Cambridge : Cambridge University Press.

Kimzin, G., & Proctor, S. (1986). An ELI academic listening comprehension needs assessment: Establishing goals, objectives, and microskills. Honolulu , HI : Unpublished scholarly paper, Department of ESL, University of Hawai ‘i at Manoa.

Long, M. H. (2005). Methodological issues in learner needs analysis in learner needs analysis. In M. H. Long (Ed.), Second Language Needs Analysis (pp. 19-76). Cambridge : Cambridge University Press.

Lynch, B. K. (1992). Evaluating a program inside and out. In J. C. Alderson & A. Beretta (Eds.), Evaluating second language education (pp. 61-99). Cambridge : Cambridge University Press.

Lynch, B. K. (2000). Evaluating a Project-Oriented CALL Innovation. Computer Assisted Language Learning, 13(4-5), 417-440.

Mackay, R., & Bosquet, M. (1981). LSP curriculum development: From policy to practice. In R. Mackay & J. D. Palmer (Eds.), Language for specific purposes: Program design and evaluation. (pp. 1-28). Rowley , MA : Newbury House.

Matthies, B., F. (1991). Administrative evaluation in ESL programs: “How’m I doin’?” In M. C. Pennington (Ed.), Building better English language programs (pp. 241- 256). Washington , DC : NAFSA.

Mitchell, R. (1989). Second language learning: Investigating the classroom context. System, 17(2), 195-210.

Pennington, M. C., & Young, A. L. (1991). Procedures and instruments for faculty evaluation in ESL. In M. C.

Pennington (Ed.), Building better English language programs: Perspectives on evaluation in ESL (pp. 191-227). Washington , DC : NAFSA.

Rea-Dickins, P., & Germaine, K. (1992). Evaluation. Oxford : Oxford University Press.

Sasaki, C. L. (1996). Teacher preferences of student behavior in Japan . JALT Journal, 18, 229-239.

Snow, M. A., & Brinton, D. M. (1988). Content-based language instruction: Investigating the effectiveness of the adjunct model. TESOL Quarterly, 22(4), 553-574.

Vandermeeren, S. (2005). Foreign language need of business firms. In M. H. Long (Ed.), Second Language Needs Analysis (pp. 159-181). Cambridge : Cambridge University Press.

Weir, C., & Roberts, J. (1994). Evaluation in ELT. Oxford : Blackwell Publishers.

Yashima, T. (2002). Willingness to communicate in L2: The Japanese EFL context. The Modern Language Journal, 86(1), 54-66.

III. Online Advice on Designing Surveys for Evaluation

Assessment Handbook

http://www.engin.umich.edu/teaching/assess_and_improve/handbook/index.html

University of Michigan College of Engineering has a comprehensive handbook for student learning outcomes assessment. In the handbook, they introduce surveys and questionnaires as one indirect measure of learning: “gather[ing] perceptions of learning, opinions about learning or reflections on learning”.

Online Survey Designers’ Guide

http://lap.umd.edu/survey_design/index.html

This technical site provides design guidelines and principles for online surveys based on a Human-Computer Interaction approach.

Resources for Survey Researchers

http://www.surveysystem.com/resource.htm
Creative Research Systems offers “Do’s and don’ts” in survey design.

Statistics: Power from Data!

http://www.cpc.unc.edu/measure/publications/index.php

Statistics Canada briefly introduces data collection methods and questionnaires.

Evaluation Handbook

http://www.ncela.gwu.edu/files/rcd/BE020502/Evaluation_Handbook.pdf

This guide to program evaluation covers planning, implementing, and reporting stages in evaluation.

Evaluation Methods

http://dmc.umn.edu/spotlight/evaluation.shtml

Evaluators at the University of Minnesota describe evaluation methods and data collection techniques. They also include some sample surveys.

Evaluate Programs with Surveys

http://www.utexas.edu/academic/diia/assessment/iar/programs/plan/method/survey.php

This useful site from the University of Texas provides a step-by-step guide to planning surveys, gathering data, analyzing data, and reporting results.

IV. Comparison of Some Online Survey Software and Features

	Free Trial & Basic	Price	Language	Support
Advanced Survey	Free: Unlimited questions and responses, but limited functions (150 characters for open-ended response)	$24.95/mo or $249.95/yr	(No double-byte characters)	Free technical support via email.
Free Assessment Summary Tool	Up to 20 questions per survey, with unlimited renewal.	FREE (At least until October, 2011.)	Not supported.	No support. One can write to the programmer.
QuestionPro	Free for 1 month, up to 100 responses. Students: Free access for 6 months, 5000 responses, unlimited questions Professors: Free access for 1yr (renewable). 5000 responses, unlimited questions. Not for funded research.	$15-$99/mo	Multiple languages.	Free technical support via email.
Student Assessment of Learning Gains	Free Limitations: Up to 250 characters per question and up to 500 characters per response.	FREE	Not supported.	No support. One can write to the programmer.
Survey Crafter	None	Licensed copy. $345-$495	No double-byte characters	Free technical support. Phone and email.
Survey Monkey	Free, but limited to 10 questions and 100 responses per survey.	$19.95/mo or $200/yr	Multiple languages.	Free technical support via email.
SurveySuite	Free for 14 days.	$69.95/yr	Not supported	No support.
Zommerang	Data stored for 10 days, limited to 30 questions, 100 responses, limited analysis	$199-$599/yr	Over 40 languages.	Free online and phone support. Good Q & A data-base.

V. Research on the Use of Online Surveys

Conn , C. (n.d.). Using the internet for surveying: techniques for designing, developing & delivering. Retrieved December 18, 2009, from the Northern Arizona University Office of Academic Assessment Website: https://www4.nau.edu/assessment/main/research/responserates.htm

Conn , C., & Norris, J. (2003). Summary Report: Investigating strategies for increasing student response rates to online-delivered course evaluations. Retrieved December 18. 2009 from http://www4.nau.edu/aio/OnlineCourseEval/ResponseRateSummary.pdf

Konstan, J. A., Rosser, B. R. S., Ross, M. W., Stanton, J., & Edwards, W. M. (2005). The story of subject naught: A cautionary but optimistic tale of Internet survey research. Journal of Computer-Mediated Communication, 10 (2), article 11. Retrieved January 24, 2006, from http://jcmc.indiana.edu/vol10/issue2/konstan.html

Norris, J. & Conn, C. (2005). Investigating strategies for increasing student response rates to online-delivered course evaluations. Quarterly Review of Distance Education, 6(1), 13-29.

Wright, R. (2005). Researching Internet-based populations: Advantages and disadvantages of online survey research, online questionnaire authoring software packages, and web survey services. Journal of Computer-Mediated Communication, 10(3), Article 11. Retrieved January 24, 2006, from http://jcmc.indiana.edu/vol10/issue3/wright.html