ON-LINE ASSESSMENT THROUGH COMMUNITIES OF PRACTICE
This qualitative study investigates whether a new framework for student assessment can be implemented in an on-line course for language teachers. The assessment framework, based on constructivist and social-interactionist theories of learning, is used in the evaluation of students as individuals in the class and as members of a community of learning. It also considers the impact that the course may have on their professional practices. In this article, the framework is explained and applied to one participant in an on-line methodology course for language teachers at the University of Hawai' i. The findings suggest that this assessment framework is a tool that may provide instructors of on-line teacher education courses with an approach that is different from traditional forms of evaluation and one that is more suited for collaborative on-line courses.
ON-LINE ASSESSMENT THROUGH COMMUNITIES OF PRACTICE
Higher education today is witnessing a change from a traditional, campus-centered, full-time system, with the teacher as the unique source of knowledge, to a new system in which students are often professional adults who cannot attend on-campus university classes. On-line courses are becoming an essential component of a university curriculum, offering a viable alternative to non-traditional students.
When dealing with new technological advances there is danger of focusing exclusively on the media, forgetting the importance of well-developed content and learner-centered activities that are well-grounded in effective teaching and learning principles (Gravette & Petersen, 2002; Petrides, 2002). Although one of the main components of any program is student evaluation, most on-line courses have limited themselves to applying the traditional face-to-face assessment techniques to the on-line environment, using tests that resemble the traditional in-class evaluation tools, e.g., the quiz tools built into commercially available on-line environments such as WebCT or Blackboard.
On-line courses require a different evaluation framework since they are based on different principles, employ a different methodology, involve different interaction patterns, make use of different skills, and follow different rules of engagement. Some on-line courses are addressing this issue by implementing alternative forms of evaluation such as collaborative assessment, problem-based assessment, self-assessment, and peer-assessment (Laurillard, 2002; O'Reilly & Morgan, 1999).
The assessment framework proposed here is based on constructivist and social-interactionist theories of learning and on the construction of a community of learning. Constructivism entails learning in a social context through group activities and collaboration in a rich learning environment that is interactive and hands-on, and promotes building on knowledge for learning how to learn (Perkins, 1991). Some researchers point out that constructivism is especially well-suited for adult learners, since they seem to learn better in situations in which they can apply their experience and previous knowledge (Huang, 2002; Knowles, Holton, & Swanson, 1998), and for on-line courses since distance learning allows students to engage in collaboration, discussion, team-work, and problem solving (Alavi, 1994).
According to Cuthell (2002), a learning community is "a large number of students and a small number of teachers, with the community supporting and developing the students' learning" (p.169). In a learning community, the goal is to advance the collective knowledge while supporting the growth of individual knowledge, emphasizing the concept of expert-apprentice relationships among participants, where some members are experts at the center of the community, and others are peripheral to the community, but still participating (Lave & Wenger, 1991).
On-line communities evolved from learning communities. They are forms of technology-mediated environments that foster a sense of community (Cuthell, 2002). Learning to successfully operate in an on-line community involves moving from the peripheral area of novice apprenticeship to the center of knowledge (Lave & Wenger, 1991). This knowledge ranges from understanding of the technology involved to the norms, rules, and etiquette of the community.
THE ASSESSMENT FRAMEWORK
One of the most popular models of on-line course assessment that considers the importance of the community of learning is that by Palloff & Pratt (1999, 2001) who propose a model based on a circular evaluation of the individual, the group, the technology, the tasks, and the facilitator.
Two components of the above model were adopted for the evaluation of students' performance in the on-line course, namely, the individual and the group. In addition, Garrison, Anderson, & Archer's (2000) "cognitive presence" or "practical inquiry" model was also incorporated. According to Garrison, Anderson, & Archer (2001), "cognitive presence is the extent to which the participants in any particular configuration of a community of inquiry are able to construct meaning through sustained reflection and discourse in a critical community of inquiry" (p. 5).
The practical inquiry model reflects four phases:
It was important that the framework accommodate a course that was conceptualized as an open entity, i.e., one that would exist not only inside but also outside the on-line environment. It was conceived as a course that would influence the students' practice and their contribution to those communities in which they are participants, such as their classrooms and schools. With this in mind, assessment needed to include a component to evaluate students as participants and co-constructors of their communities of practice, and as individuals with changing identities who could trigger improvements around them, particularly in their teaching. For this reason, the fourth phase in Garrison, Anderson, & Archer's model is decisive, since it helps evaluate the impact of the on-line course on the participants' professional practices.
METHODOLOGY: APPLYING THE FRAMEWORK
This study investigated whether the assessment framework presented above can be implemented in an actual on-line course. A qualitative approach was selected since data was based on the perceptions of the people in the environment, and individuals were the primary collection instrument (Key, 1997). In addition, this study can be considered participatory research (Bogdan & Bikle, 1998) since the researcher/instructor was a participant observer in the on-line course.
The framework presented above was used to evaluate students enrolled in an on-line methods course for language teachers at the University of Hawai' i. It was developed for pre- and in-service teachers who wanted to acquire or update their knowledge of methodology of second language learning, new techniques and research in the field, as well as application of new technologies to language teaching. Given the geographical isolation of the Hawai'ian islands, the course was conducted on-line to attract participants from different islands. The main goal of the course was to promote collaboration between pre- and in-service teachers, taking advantage of the multiple and dynamic novice-expert levels of knowledge of the participants.
The course consisted of three main units each containing several chapters. Each unit included one or more readings in PDF format preceded by a presentation of the topic and followed by a reflective asynchronous discussion in an electronic forum, similar to a bulletin board. Some units also included a project in which the participants had to apply new knowledge to create artifacts for future use in their classrooms, plus an essay describing and reflecting on the design, process, and product of the project. In addition, there was a chat-room for synchronous communication where students met to interact, ask questions, or visit the instructor during virtual office hours. The on-line environment hosting the methodology course integrated several interactive tools such as a discussion forum with audio capability, a chat-room with audio and video capability, and uploading and downloading of shared documents, all in a password-protected environment. It also had the capacity to produce detailed information on students' online activities, as well as generate grade reports.
Participants ranged from expert teachers with 15 or more years of teaching experience to pre-service teachers with no teaching experience at all. Some teachers were experienced educators with little knowledge of technology, while others were very comfortable with technology, but were apprentices in the matter of course content. All participants, except one, were teaching a language course while participating in the on-line methods course. The levels they taught ranged from kindergarten to university. One participant was teaching four-year-olds at Sunday school.
Data Collection and Analysis
Several sources and methods were employed to gather the data because multiple sources and methods allow for triangulation of data and produce more balanced and reliable results (Jasso-Aguilar, 1999; Key, 1997). The main sources of data were participants' discussion forum entries, e-mail messages, and projects, particularly the reflective component of the latter. These included self-reflection, observation, questionnaires, and technical reports about students' behavior in the on-line environment. Table 1 presents the methods and sources used to evaluate the students as individuals, as part of the community of learning, and according to the impact of the course on the participants.
Measuring concepts such as "sense of accomplishment" or "satisfaction with the process" was carried out qualitatively by assigning either "++" sign to an "always" or "most of the time" condition, "+" to "sometimes" condition, and "-" to a "hardly ever" or "never" condition (Hatch & Lazaraton, 1991).
Table 1. Methods and Sources of Evaluation
Students as individuals
Students' self-assessment paragraphs in the discussion forums and in their unit projects were analyzed following Palloff and Pratt's model (1991, 2001). These paragraphs described the students' sense of accomplishment and satisfaction with the creative process. E-mail messages to the instructor were also analyzed for references to students' level of satisfaction with the course and with their own participation.
The course instructor assessed the quality of student products using different rubrics. See Appendix A for an example of a rubric adapted from Shrum & Glisan (2000) that was used to assess the projects. Students' ability to work at their own pace was assessed through reports generated by the course software. These reports provided information on the frequency and duration of each student's presence in the online environment. The rubrics also provided information about the students' activities while in the course environment, such as number and length of postings, answers to other students' postings, initiation of forum discussions, and uploads. An end-of-semester questionnaire was used to assess students' sense of self-expression and reflection .
Students as part of the community
Several sources and methods were used to evaluate students as part of the learning community. Garrison, Anderson, & Archer' s (2000) practical inquiry model was followed to evaluate the students' shared knowledge and co-construction of meaning. Accordingly, students' forum entries and e-mail communications were analyzed to identify whether they had engaged in the four phases of the model. Following the same criteria as above, "++" was assigned to an "always" or "most of the time" condition, "+" was assigned to "sometimes" condition, and "-" was assogmed to a "hardly ever" or "never" condition. Students' artifacts and task products, mainly from their lesson projects, were also evaluated. The discussion forums, e-mail messages exchanged between students and instructor, conversations in the chat-room, and the reflection paragraphs in their unit projects were the main sources of information. Self-assessment was used as the main method of collecting data about students' identity change and sense of belonging to a community as well as their evolution in the community from a peripheral novice participant to a more central and expert one.
Impact of the course
The third part of the assessment framework focused on the impact of the course on the participants' practices in their communities of learning outside of class. To evaluate this impact, it was necessary to move from the on-line environment to the participants' places of practice. The participants' classes were observed and a questionnaire was developed for distribution among the participants' students and co-workers. Due to unforeseen circumstances (a flood) these evaluations became impossible to implement. As a result, data about the impact of the course on participants' teaching practices is based on their "resolutions" to implement the newly acquired knowledge. The data were gathered through self-assessment paragraphs, final course evaluation, and through forum entries and e-mail messages.
The instructor/researcher performed all the rating as part of the evaluation and grade assignment for the course. For purposes of establishing the reliability of instructor ratings (Stainback & Stainback, 1988), an external evaluator examined ten random entries. Inter-rater agreement was 0.9 (9 out of 10 entries). The only disagreement resulted in a slight adjustment of the criteria.
Applying the framework to evaluate Jane
To see if the framework is a viable evaluative tool, I applied it to one of the participants whom we will call Jane (not her real name) by analyzing data from her forum entries, e-mail messages, and projects. Jane is not a native speaker of English, so her entries reflect her imperfect command of the language.
As an individual participant, Jane' s sense of accomplishment and satisfaction with the on-line course process was evident from the reflection part of her projects, her forum entries, and e-mail communications:
"++" was assigned to this category since she expressed a sense of accomplishment repeatedly. As for her sense of self-expression and reflection, Jane also received "++", since her projects always incorporated a carefully considered reflective paragraph. They averaged an "A" according to the project assessment criteria used to evaluate tasks and artifacts (Appendix A) because they included a clear description of the activity, set attainable and measurable objectives, and provided materials to support them. They also showed clear evidence of reflection on the unit content through the readings and forum discussions (see Appendix A for full criteria).
Jane completed all assignments and posted her comments in a relatively timely manner. Therefore, she could receive comments from other members of the community, and they could benefit from her postings. Since Jane is a non-native speaker of English, she felt linguistically disadvantaged and often skipped a few entries before posting hers, so that she could read other students' entries and feel more comfortable with own postings. The following statements are from a chat session and one of her forum entries.
"I am still trying to improve my English."
"Sometimes it's really hard for me when everybody seems to know everything and I am still trying digesting the topics. However I am going to try my best to catch up with the others." (from chat-room during virtual office hours)
"Thank you very much for your explanation [responding to another student]. I already read the other participants' comment too and I think I have a better understanding about CoP or CoL."(from Forum Unit 1)
As part of the community of learning, Jane' s forum entries were coded according to Garrison, Anderson, & Archer's (2000) practical inquiry model by looking at whether she engaged in the four phases of the model.Phase 1: Triggering Event
Jane engaged in this phase very often by raising numerous questions about the topics and the assigned readings.
"I don't really understand about inner-speech. I know that in ZPD inner-speech is an important tool of thought. Is it a kind of self-awareness when we are thinking using the target language? Is it a kind of monologue with ourselves?" (from Forum Unit 2)
Phase 2: Exploration
In this phase, Jane exchanged questions and opinions with the other members of the community. This helped her to explore the topic in greater depth. The example below shows how Jane is starting to volunteer her opinion more freely. She even provides examples of her own practice, offering personal information.
Phase 3: Integration
Jane actively participated in this phase too. She formed her ideas based on the readings, the discussion, and her own experience and knowledge.
"I agree with the idea that students can acquire language in low anxiety situation. I think in every field of study we can't really learn something under preassure. So this is a good thing about NA." (from a reflective paragraph project # 3)
Regarding Jane' s role and identity in the community, we have already seen that as a non-native speaker of English she often commented on her difficulties with the language. She also thought of herself as a novice in the community, even though she had teaching experience at the university level.
"So far, I think I am definitely a peripheral participant because I don' t share my ideas a lot and I just read a lot of other participants' comment. I really hope that I am going to be a full participant (especially for this on-line course) who not only take something but also give something to the community." (from an e-mail communication with instructor).
Although Jane thought of herself as a novice and peripheral member of the community, she was considerably more expert in some of the applications of technology than others. The following is an example from the chat-room in which Jane is talking to Mary (not her real name), a more experienced teacher but one who has never used chat-rooms before. Jane knows how chat-rooms work, she defines herself as a long-time user, and she even uses discourse that is typical of computer-mediated communication, such as the abbreviation "ppl" for "people", and the emoticon ":)" for a smile.
(from Chat-room Unit 2)
Towards the end of the semester, Jane stopped talking about herself as a novice and started contributing as a central member of the community, moving from the use of "I" as an individual to the use of "we" as an integral part of the group. She even placed herself at the level of April (not her real name), the most advanced and experienced student in the class.
(from Forum Unit 3)
"In conclusion, I don't have specific preference of which approach I should use in my class. The same as April, I never use one approach exclusively in my class. I usually combine some of them."(from Discussion Forum Unit 3)
From a discourse analysis of Jane' s forum entries, e-mail messages, and chat interactions, she was assigned "++" for the amount of knowledge she shared and for her contribution to the co-construction of community knowledge. She also received "++" for her evolution from a peripheral position in the community to a more central one, and "++" for her sense of community and own identity.
The last item in the framework, the impact of the course on Jane' s language teaching practices, was not assessed as planned because of the unforeseen event mentioned above. Nevertheless, information about Jane's intention to use some of her newly-acquired knowledge in her practice could still be gathered from Phase 4 of the inquiry model which focuses on the participants' resolution to implement new knowledge into practice. Although previous research has shown that phase 4 is unusual in community interaction (Pawan, Paulus, Yalcin, & Chang, 2003), Jane' s entries illustrate several instances of her resolution to apply her new knowledge to her teaching.
"I think applying WebQuest in my class will be very interesting. I will talk to my supervisor about this. Who knows I can implement this thing next semester." (from Forum Unit 2).
"I am sure that these criteria of assessment have so many limitations. It is definitely not a perfect work yet and I need some suggestions to improve it. I am also not so sure if it can be implemented in class X but I am going to try to use it for my class. By doing so, I will know whether the assessment criteria that I made really work or not." (from Project #4)
Jane received "++" for her intention to implement the course content into her practice, although the real impact of the course could not be evaluated.
DISCUSSION AND CONCLUSION
As more institutions incorporate on-line courses in their curriculum, educators need to remember that the medium of distribution and the organization of the course affect and shape not only its content, but also the way in which the students are evaluated. Traditional forms of assessment, such as quizzes and exams, are limited tools that can give only a partial picture of students' performance in on-line courses, especially if their goal is the creation and distribution of co-constructed knowledge.
The main focus of this paper was to present an assessment framework that considers an on-line course as a community of learning, an interactive environment in which knowledge is created and shared. In this environment, students are evaluated from three different perspectives: first, as individuals who contribute to the creation of knowledge, second, as members of the community who help co-construct knowledge through their participation, and finally, the framework considers the impact of the course on the participants' professional practices. The study has demonstrated that it is possible to implement this framework, using Jane, a participant in an on-line course for language teachers at the University of Hawai' i, as a model. Jane' s e-mail messages, her forum entries, and her projects were examined qualitatively to assign a holistic grade to each of the assessment components.
It is important to note that the assessment framework was developed for one specific course. It needs to be tested in other similar courses and with different students, allowing for the possibility that it may be combined with other forms of assessment. It is ultimately up to the instructors to decide what form of assessment will provide them with the best information about their students' progress and growth. Assessment in on-line courses is still a developing field, and as the demand for these courses increases, so should research.
ABOUT THE AUTHOR
Marta Gonzalez-Lloret is an instructor in the Spanish department at the University of Hawai' i. She is also a PhD candidate in the department of Second Language Studies at the University of Hawai' i. Her interests include uses of CALL and assessment of on-line environments for language teaching.
APPENDIX A. Rubric for Project Assessment (adapted from Shrum & Glisan, 2000)
Alavi, M. (1994). Computer-mediated collaborative learning: An empirical evaluation. MIS Quarterly, 18(2), 159-174.
Bogdan, R., & Biklen, S. K. (1998). Qualitative research in education: An introduction to theory and methods. Needham, MA: Allyn & Bacon.
Cuthell, J. (2002). MirandaNet: A learning community-a community of learners. Journal of Interactive Learning Research, 13(1/2), 167-186.
Garrison, R., Anderson, T., & Archer, W. (2001). Critical thinking, cognitive presence, and computer conferencing in distance education. American Journal of Distance Education, 15(1), 7-23.
Garrison, R., Anderson, T., & Archer, W. (2000). Critical thinking in a text-based environment: Computer conferencing in higher education. Internet and Higher Education, 11(2), 1-14.
Gravett, S., & Petersen, N. (2002). Structuring dialogue with students via learning tasks. Innovative Higher Education, 2(4), 281-291.
Hatch, E., & Lazaraton, A. (1991). The research manual: Design and statistics for applied linguistics. New York, NY: Newbury House Publishers.
Huang, H-M. (2002). Towards constructivism for adult learners in on-line environments. British Journal of Educational Technology, 33(1), 27-37.
Jasso-Aguilar, R. (1999). Sources, methods and triangulation in needs analysis: A critical perspective in a case study of Waikiki hotel maids. English for Specific Purposes, 18(1), 27-46.Key, J. P. (1997). Research design in occupational education. Retrieved May 15, 2005 from http://www.okstate.edu/ag/agedcm4h/academic/aged5980a/5980/newpage2.htm
Knowles, M., Holton, E., & Swanson, R. (1998). The adult learner: The definitive classic in adult education and human resource development (5 th ed.). Houston, TX: Gulf Publishing.
Laurillard, D. (2000). Rethinking university teaching: A conversational framework for the effective use of educational technology. London: Routledge.
Lave, J., & Wenger, E. (1991). Situated learning: Legitimate peripheral participation. Cambridge [ England]; New York: Cambridge University Press.
O' Reilly, M., & Morgan, C. (1999). On-line assessment: Creating communities and opportunities. In S. Brown, P. Race, & J. Bull, J. (Eds.), Computer-assisted assessment in Higher Education (pp. 149-161). London: Kogan Page.
Palloff, R., & Pratt, K. (1999) Building learning communities in cyberspace. San Francisco, CA: Jossey-Bass.
Palloff, R., & Pratt, K. (2001). Lessons from the cyberspace classroom. San Francisco, CA: Jossey-Bass.Pawan, F., Paulus, T.M., Yalcin, S., & Chang, C.-F. (2003). Online learning: Patterns of engagement and interaction among in-service teachers. Language Learning & Technology, 7(3), 119-140. Retrieved September 5, 2005 from http://llt.msu.edu/vol7num3/pawan/default.html.
Perkins, (1991). Technology meets constructivism: Do they make a marriage? Educational Technology, 31(5), 18-23.
Petrides, L. (2002). Web-based technology for distributed (or distance) learning: Creating learning-centered educational experiences in the higher education classroom. International Journal of Instructional Media, 29(1), 69-77.
Shrum, J. L., & Glisan, E. W. (2000). Teacher's handbook: Contextualized language instruction, (2nd ed.). Boston: Thomson Learning.
Stainback, S., & Stainback, W. (1988). Understanding & conducting qualitative research . Dubuque, Iowa: Kendall/Hunt.
Vygotsky, L. (1986). Thought and language. Cambridge, MA: MIT Press.
Wallace, M. J. (1998). Action research for language teachers. Cambridge, England: Cambridge University Press.
Wenger, E. (1998). Communities of practice: Learning, meaning, and identity. Cambridge, U.K; New York, N.Y: Cambridge University Press.
Copyright © 2005 National Foreign Language Resource Center
Articles are copyrighted by their respective authors.