ON-LINE ASSESSMENT THROUGH COMMUNITIES OF PRACTICE

ON-LINE ASSESSMENT THROUGH COMMUNITIES OF PRACTICE

Marta Gonzalez-Lloret
University of Hawai'i, Manoa

ABSTRACT

This qualitative study investigates whether a new framework for student assessment can be implemented in an on-line course for language teachers. The assessment framework, based on constructivist and social-interactionist theories of learning, is used in the evaluation of students as individuals in the class and as members of a community of learning. It also considers the impact that the course may have on their professional practices. In this article, the framework is explained and applied to one participant in an on-line methodology course for language teachers at the University of Hawai' i. The findings suggest that this assessment framework is a tool that may provide instructors of on-line teacher education courses with an approach that is different from traditional forms of evaluation and one that is more suited for collaborative on-line courses.

ON-LINE ASSESSMENT THROUGH COMMUNITIES OF PRACTICE

Higher education today is witnessing a change from a traditional, campus-centered, full-time system, with the teacher as the unique source of knowledge, to a new system in which students are often professional adults who cannot attend on-campus university classes. On-line courses are becoming an essential component of a university curriculum, offering a viable alternative to non-traditional students.

When dealing with new technological advances there is danger of focusing exclusively on the media, forgetting the importance of well-developed content and learner-centered activities that are well-grounded in effective teaching and learning principles (Gravette & Petersen, 2002; Petrides, 2002). Although one of the main components of any program is student evaluation, most on-line courses have limited themselves to applying the traditional face-to-face assessment techniques to the on-line environment, using tests that resemble the traditional in-class evaluation tools, e.g., the quiz tools built into commercially available on-line environments such as WebCT or Blackboard.

On-line courses require a different evaluation framework since they are based on different principles, employ a different methodology, involve different interaction patterns, make use of different skills, and follow different rules of engagement. Some on-line courses are addressing this issue by implementing alternative forms of evaluation such as collaborative assessment, problem-based assessment, self-assessment, and peer-assessment (Laurillard, 2002; O'Reilly & Morgan, 1999).

The assessment framework proposed here is based on constructivist and social-interactionist theories of learning and on the construction of a community of learning. Constructivism entails learning in a social context through group activities and collaboration in a rich learning environment that is interactive and hands-on, and promotes building on knowledge for learning how to learn (Perkins, 1991). Some researchers point out that constructivism is especially well-suited for adult learners, since they seem to learn better in situations in which they can apply their experience and previous knowledge (Huang, 2002; Knowles, Holton, & Swanson, 1998), and for on-line courses since distance learning allows students to engage in collaboration, discussion, team-work, and problem solving (Alavi, 1994).

According to Cuthell (2002), a learning community is "a large number of students and a small number of teachers, with the community supporting and developing the students' learning" (p.169). In a learning community, the goal is to advance the collective knowledge while supporting the growth of individual knowledge, emphasizing the concept of expert-apprentice relationships among participants, where some members are experts at the center of the community, and others are peripheral to the community, but still participating (Lave & Wenger, 1991).

On-line communities evolved from learning communities. They are forms of technology-mediated environments that foster a sense of community (Cuthell, 2002). Learning to successfully operate in an on-line community involves moving from the peripheral area of novice apprenticeship to the center of knowledge (Lave & Wenger, 1991). This knowledge ranges from understanding of the technology involved to the norms, rules, and etiquette of the community.

THE ASSESSMENT FRAMEWORK

One of the most popular models of on-line course assessment that considers the importance of the community of learning is that by Palloff & Pratt (1999, 2001) who propose a model based on a circular evaluation of the individual, the group, the technology, the tasks, and the facilitator.

The individuals are assessed according to their own sense of accomplishment, quality of outcome, satisfaction with the process, ability to work at their own pace, and sense of self-expression.
The group is assessed on its collaboration, teamwork, sense of well-being, support, reflection, and reduced isolation.
The evaluation of technology includes looking at the communication tools, tasks, and the transparency and ease of use.
The tasks are assessed according to three criteria: a common sense of purpose, whether they are a source of motivation, and whether they promote collaboration among the other participants.
The facilitator is evaluated on factors such as clear and balanced communication, creation of a safe environment for interaction, promotion of relationships, and encouragement of students' self-organization and empowerment.

Two components of the above model were adopted for the evaluation of students' performance in the on-line course, namely, the individual and the group. In addition, Garrison, Anderson, & Archer's (2000) "cognitive presence" or "practical inquiry" model was also incorporated. According to Garrison, Anderson, & Archer (2001), "cognitive presence is the extent to which the participants in any particular configuration of a community of inquiry are able to construct meaning through sustained reflection and discourse in a critical community of inquiry" (p. 5).

The practical inquiry model reflects four phases:

Initiation in which a triggering event such as a task, a teacher's question, or a problem to be solved starts the dialogue.
Exploration in which students move from private to social exploration of ideas. It is usually characterized by brainstorming, questions, and dialogue.
Integration in which students start producing ideas as a result of the exploration phase, and start "constructing meaning."
Resolution in which students apply their knowledge to explicit actions. In this phase students implement and test their new knowledge - a process that may take them back to the first phase.

It was important that the framework accommodate a course that was conceptualized as an open entity, i.e., one that would exist not only inside but also outside the on-line environment. It was conceived as a course that would influence the students' practice and their contribution to those communities in which they are participants, such as their classrooms and schools. With this in mind, assessment needed to include a component to evaluate students as participants and co-constructors of their communities of practice, and as individuals with changing identities who could trigger improvements around them, particularly in their teaching. For this reason, the fourth phase in Garrison, Anderson, & Archer's model is decisive, since it helps evaluate the impact of the on-line course on the participants' professional practices.

METHODOLOGY: APPLYING THE FRAMEWORK

This study investigated whether the assessment framework presented above can be implemented in an actual on-line course. A qualitative approach was selected since data was based on the perceptions of the people in the environment, and individuals were the primary collection instrument (Key, 1997). In addition, this study can be considered participatory research (Bogdan & Bikle, 1998) since the researcher/instructor was a participant observer in the on-line course.

The Course

The framework presented above was used to evaluate students enrolled in an on-line methods course for language teachers at the University of Hawai' i. It was developed for pre- and in-service teachers who wanted to acquire or update their knowledge of methodology of second language learning, new techniques and research in the field, as well as application of new technologies to language teaching. Given the geographical isolation of the Hawai'ian islands, the course was conducted on-line to attract participants from different islands. The main goal of the course was to promote collaboration between pre- and in-service teachers, taking advantage of the multiple and dynamic novice-expert levels of knowledge of the participants.

The course consisted of three main units each containing several chapters. Each unit included one or more readings in PDF format preceded by a presentation of the topic and followed by a reflective asynchronous discussion in an electronic forum, similar to a bulletin board. Some units also included a project in which the participants had to apply new knowledge to create artifacts for future use in their classrooms, plus an essay describing and reflecting on the design, process, and product of the project. In addition, there was a chat-room for synchronous communication where students met to interact, ask questions, or visit the instructor during virtual office hours. The on-line environment hosting the methodology course integrated several interactive tools such as a discussion forum with audio capability, a chat-room with audio and video capability, and uploading and downloading of shared documents, all in a password-protected environment. It also had the capacity to produce detailed information on students' online activities, as well as generate grade reports.

The Participants

Participants ranged from expert teachers with 15 or more years of teaching experience to pre-service teachers with no teaching experience at all. Some teachers were experienced educators with little knowledge of technology, while others were very comfortable with technology, but were apprentices in the matter of course content. All participants, except one, were teaching a language course while participating in the on-line methods course. The levels they taught ranged from kindergarten to university. One participant was teaching four-year-olds at Sunday school.

Data Collection and Analysis

Several sources and methods were employed to gather the data because multiple sources and methods allow for triangulation of data and produce more balanced and reliable results (Jasso-Aguilar, 1999; Key, 1997). The main sources of data were participants' discussion forum entries, e-mail messages, and projects, particularly the reflective component of the latter. These included self-reflection, observation, questionnaires, and technical reports about students' behavior in the on-line environment. Table 1 presents the methods and sources used to evaluate the students as individuals, as part of the community of learning, and according to the impact of the course on the participants.

Measuring concepts such as "sense of accomplishment" or "satisfaction with the process" was carried out qualitatively by assigning either "++" sign to an "always" or "most of the time" condition, "+" to "sometimes" condition, and "-" to a "hardly ever" or "never" condition (Hatch & Lazaraton, 1991).

Table 1. Methods and Sources of Evaluation

Assessment categories	Methods	Sources
As an individual
Sense of accomplishment	Self-reflection	Projects discussion forum
Quality of contributions	Qualitative rubric	Teacher's grade
Satisfaction with the process	Self-reflection	E-mail, projects, discussion forum
Ability to work at one' s own pace	On-line course records	Server files
Sense of self-expression and reflection	Self-assessment	Final evaluation/questionnaire
As part of the learning community
Shared knowledge and co-construction of meaning	Practical inquiry model Evaluation of tasks, artifacts Self-reflection	Projects Discussion forum Project assessment criteria
Expert-novice relationship	Self-reflection	E-mail Discussion forum entries Chat room
Identity and sense of community	Self-reflection	Forum entries Projects

Students as individuals

Students' self-assessment paragraphs in the discussion forums and in their unit projects were analyzed following Palloff and Pratt's model (1991, 2001). These paragraphs described the students' sense of accomplishment and satisfaction with the creative process. E-mail messages to the instructor were also analyzed for references to students' level of satisfaction with the course and with their own participation.

The course instructor assessed the quality of student products using different rubrics. See Appendix A for an example of a rubric adapted from Shrum & Glisan (2000) that was used to assess the projects. Students' ability to work at their own pace was assessed through reports generated by the course software. These reports provided information on the frequency and duration of each student's presence in the online environment. The rubrics also provided information about the students' activities while in the course environment, such as number and length of postings, answers to other students' postings, initiation of forum discussions, and uploads. An end-of-semester questionnaire was used to assess students' sense of self-expression and reflection .

Students as part of the community

Several sources and methods were used to evaluate students as part of the learning community. Garrison, Anderson, & Archer' s (2000) practical inquiry model was followed to evaluate the students' shared knowledge and co-construction of meaning. Accordingly, students' forum entries and e-mail communications were analyzed to identify whether they had engaged in the four phases of the model. Following the same criteria as above, "++" was assigned to an "always" or "most of the time" condition, "+" was assigned to "sometimes" condition, and "-" was assogmed to a "hardly ever" or "never" condition. Students' artifacts and task products, mainly from their lesson projects, were also evaluated. The discussion forums, e-mail messages exchanged between students and instructor, conversations in the chat-room, and the reflection paragraphs in their unit projects were the main sources of information. Self-assessment was used as the main method of collecting data about students' identity change and sense of belonging to a community as well as their evolution in the community from a peripheral novice participant to a more central and expert one.

Impact of the course

The third part of the assessment framework focused on the impact of the course on the participants' practices in their communities of learning outside of class. To evaluate this impact, it was necessary to move from the on-line environment to the participants' places of practice. The participants' classes were observed and a questionnaire was developed for distribution among the participants' students and co-workers. Due to unforeseen circumstances (a flood) these evaluations became impossible to implement. As a result, data about the impact of the course on participants' teaching practices is based on their "resolutions" to implement the newly acquired knowledge. The data were gathered through self-assessment paragraphs, final course evaluation, and through forum entries and e-mail messages.

The instructor/researcher performed all the rating as part of the evaluation and grade assignment for the course. For purposes of establishing the reliability of instructor ratings (Stainback & Stainback, 1988), an external evaluator examined ten random entries. Inter-rater agreement was 0.9 (9 out of 10 entries). The only disagreement resulted in a slight adjustment of the criteria.

Applying the framework to evaluate Jane

To see if the framework is a viable evaluative tool, I applied it to one of the participants whom we will call Jane (not her real name) by analyzing data from her forum entries, e-mail messages, and projects. Jane is not a native speaker of English, so her entries reflect her imperfect command of the language.

As an individual participant, Jane' s sense of accomplishment and satisfaction with the on-line course process was evident from the reflection part of her projects, her forum entries, and e-mail communications:

"I am satisfied with the process of making this assessment. I am also quite satisfied with the product that I make. I am sure that I can still do a lot to improve my mini project but I do not have more time so I just feel satisfied with what I already did for a short time." (from reflection section of project #2)

"Despite the imperfectness in my project, I am satisfied with the process that I' ve been through in making this project. As I mentioned at the beginning of my reflection that this is the first time for me to make assessment criteria so it is a great experience for me. I am not only learning the theory of assessment but I also do the real thing by making this project." (from an e-mail communication with the instructor)

"++" was assigned to this category since she expressed a sense of accomplishment repeatedly. As for her sense of self-expression and reflection, Jane also received "++", since her projects always incorporated a carefully considered reflective paragraph. They averaged an "A" according to the project assessment criteria used to evaluate tasks and artifacts (Appendix A) because they included a clear description of the activity, set attainable and measurable objectives, and provided materials to support them. They also showed clear evidence of reflection on the unit content through the readings and forum discussions (see Appendix A for full criteria).

Jane completed all assignments and posted her comments in a relatively timely manner. Therefore, she could receive comments from other members of the community, and they could benefit from her postings. Since Jane is a non-native speaker of English, she felt linguistically disadvantaged and often skipped a few entries before posting hers, so that she could read other students' entries and feel more comfortable with own postings. The following statements are from a chat session and one of her forum entries.

As part of the community of learning, Jane' s forum entries were coded according to Garrison, Anderson, & Archer's (2000) practical inquiry model by looking at whether she engaged in the four phases of the model.

Phase 1: Triggering Event

Jane engaged in this phase very often by raising numerous questions about the topics and the assigned readings.

Phase 2: Exploration

In this phase, Jane exchanged questions and opinions with the other members of the community. This helped her to explore the topic in greater depth. The example below shows how Jane is starting to volunteer her opinion more freely. She even provides examples of her own practice, offering personal information.

"I agree with Alice and Ana that internet is a really good source. I usually search the materials for my classes from the internet. I also use magazines and news paper. Some of the time I use songs and movies too.
(From Forum Unit 2)

Phase 3: Integration

Jane actively participated in this phase too. She formed her ideas based on the readings, the discussion, and her own experience and knowledge.

Regarding Jane' s role and identity in the community, we have already seen that as a non-native speaker of English she often commented on her difficulties with the language. She also thought of herself as a novice in the community, even though she had teaching experience at the university level.

Although Jane thought of herself as a novice and peripheral member of the community, she was considerably more expert in some of the applications of technology than others. The following is an example from the chat-room in which Jane is talking to Mary (not her real name), a more experienced teacher but one who has never used chat-rooms before. Jane knows how chat-rooms work, she defines herself as a long-time user, and she even uses discourse that is typical of computer-mediated communication, such as the abbreviation "ppl" for "people", and the emoticon ":)" for a smile.

Mary:	I've just finished classes and conferencing with a student. I've never done a live chat before and am not quite sure how this works.
Jane:	Type on the lower window, hit enter or click send
Mary:	Do you use chat-room often/
Jane	Yeah when I was in high school

(from Chat-room Unit 2)

Towards the end of the semester, Jane stopped talking about herself as a novice and started contributing as a central member of the community, moving from the use of "I" as an individual to the use of "we" as an integral part of the group. She even placed herself at the level of April (not her real name), the most advanced and experienced student in the class.

From a discourse analysis of Jane' s forum entries, e-mail messages, and chat interactions, she was assigned "++" for the amount of knowledge she shared and for her contribution to the co-construction of community knowledge. She also received "++" for her evolution from a peripheral position in the community to a more central one, and "++" for her sense of community and own identity.

The last item in the framework, the impact of the course on Jane' s language teaching practices, was not assessed as planned because of the unforeseen event mentioned above. Nevertheless, information about Jane's intention to use some of her newly-acquired knowledge in her practice could still be gathered from Phase 4 of the inquiry model which focuses on the participants' resolution to implement new knowledge into practice. Although previous research has shown that phase 4 is unusual in community interaction (Pawan, Paulus, Yalcin, & Chang, 2003), Jane' s entries illustrate several instances of her resolution to apply her new knowledge to her teaching.

Jane received "++" for her intention to implement the course content into her practice, although the real impact of the course could not be evaluated.

DISCUSSION AND CONCLUSION

As more institutions incorporate on-line courses in their curriculum, educators need to remember that the medium of distribution and the organization of the course affect and shape not only its content, but also the way in which the students are evaluated. Traditional forms of assessment, such as quizzes and exams, are limited tools that can give only a partial picture of students' performance in on-line courses, especially if their goal is the creation and distribution of co-constructed knowledge.

The main focus of this paper was to present an assessment framework that considers an on-line course as a community of learning, an interactive environment in which knowledge is created and shared. In this environment, students are evaluated from three different perspectives: first, as individuals who contribute to the creation of knowledge, second, as members of the community who help co-construct knowledge through their participation, and finally, the framework considers the impact of the course on the participants' professional practices. The study has demonstrated that it is possible to implement this framework, using Jane, a participant in an on-line course for language teachers at the University of Hawai' i, as a model. Jane' s e-mail messages, her forum entries, and her projects were examined qualitatively to assign a holistic grade to each of the assessment components.

It is important to note that the assessment framework was developed for one specific course. It needs to be tested in other similar courses and with different students, allowing for the possibility that it may be combined with other forms of assessment. It is ultimately up to the instructors to decide what form of assessment will provide them with the best information about their students' progress and growth. Assessment in on-line courses is still a developing field, and as the demand for these courses increases, so should research.

ABOUT THE AUTHOR

Marta Gonzalez-Lloret is an instructor in the Spanish department at the University of Hawai' i. She is also a PhD candidate in the department of Second Language Studies at the University of Hawai' i. Her interests include uses of CALL and assessment of on-line environments for language teaching.

e-mail: marta@hawaii.edu
http://marta.LLL.hawaii.edu

APPENDIX A. Rubric for Project Assessment (adapted from Shrum & Glisan, 2000)

Exceeds = Meets = Below =		A - A+ C - B+ F - C -
Overview
Comprehensive class description usable to plan instruction	Sufficient class description	Insufficient class description
Project
Clear description of activity	Unit has unifying theme(s)	Unit lacks unifying theme(s)
Objectives stated in terms of measurable student outcomes	Most objectives stated in forms of measurable student outcomes	Vague objectives
Clear evidence of reflection on unit topic	Project addresses topic in class unit	Does not adequately address unit topic
Clear evidence of reflection on issues discussed on unit forum	Project addresses issues discussed in forum interaction	Does not adequately address unit topic
Ample evidence that materials and activities support objectives	Sufficient evidence that materials and activities support objectives	Insufficient examples, relationship to lessons' concepts is not clear.
Reflections
Defends teacher' s choices convincingly.	Defends teacher' s choices.	Does not effectively defend choices.
Clear reflection on the process.	Some reflection on the process.	No reflection on the process.
Clear rationale for future use.	Mention of future use.	No mention of future use.

REFERENCES

Alavi, M. (1994). Computer-mediated collaborative learning: An empirical evaluation. MIS Quarterly, 18(2), 159-174.

Bogdan, R., & Biklen, S. K. (1998). Qualitative research in education: An introduction to theory and methods. Needham, MA: Allyn & Bacon.

Cuthell, J. (2002). MirandaNet: A learning community-a community of learners. Journal of Interactive Learning Research, 13(1/2), 167-186.

Garrison, R., Anderson, T., & Archer, W. (2001). Critical thinking, cognitive presence, and computer conferencing in distance education. American Journal of Distance Education, 15(1), 7-23.

Garrison, R., Anderson, T., & Archer, W. (2000). Critical thinking in a text-based environment: Computer conferencing in higher education. Internet and Higher Education, 11(2), 1-14.

Gravett, S., & Petersen, N. (2002). Structuring dialogue with students via learning tasks. Innovative Higher Education, 2(4), 281-291.

Hatch, E., & Lazaraton, A. (1991). The research manual: Design and statistics for applied linguistics. New York, NY: Newbury House Publishers.

Huang, H-M. (2002). Towards constructivism for adult learners in on-line environments. British Journal of Educational Technology, 33(1), 27-37.

Jasso-Aguilar, R. (1999). Sources, methods and triangulation in needs analysis: A critical perspective in a case study of Waikiki hotel maids. English for Specific Purposes, 18(1), 27-46.

Key, J. P. (1997). Research design in occupational education. Retrieved May 15, 2005 from http://www.okstate.edu/ag/agedcm4h/academic/aged5980a/5980/newpage2.htm

Knowles, M., Holton, E., & Swanson, R. (1998). The adult learner: The definitive classic in adult education and human resource development (5 th ed.). Houston, TX: Gulf Publishing.

Laurillard, D. (2000). Rethinking university teaching: A conversational framework for the effective use of educational technology. London: Routledge.

Lave, J., & Wenger, E. (1991). Situated learning: Legitimate peripheral participation. Cambridge [ England]; New York: Cambridge University Press.

O' Reilly, M., & Morgan, C. (1999). On-line assessment: Creating communities and opportunities. In S. Brown, P. Race, & J. Bull, J. (Eds.), Computer-assisted assessment in Higher Education (pp. 149-161). London: Kogan Page.

Palloff, R., & Pratt, K. (1999) Building learning communities in cyberspace. San Francisco, CA: Jossey-Bass.

Palloff, R., & Pratt, K. (2001). Lessons from the cyberspace classroom. San Francisco, CA: Jossey-Bass.

Pawan, F., Paulus, T.M., Yalcin, S., & Chang, C.-F. (2003). Online learning: Patterns of engagement and interaction among in-service teachers. Language Learning & Technology, 7(3), 119-140. Retrieved September 5, 2005 from http://llt.msu.edu/vol7num3/pawan/default.html.

Perkins, (1991). Technology meets constructivism: Do they make a marriage? Educational Technology, 31(5), 18-23.

Petrides, L. (2002). Web-based technology for distributed (or distance) learning: Creating learning-centered educational experiences in the higher education classroom. International Journal of Instructional Media, 29(1), 69-77.

Shrum, J. L., & Glisan, E. W. (2000). Teacher's handbook: Contextualized language instruction, (2nd ed.). Boston: Thomson Learning.

Stainback, S., & Stainback, W. (1988). Understanding & conducting qualitative research . Dubuque, Iowa: Kendall/Hunt.

Vygotsky, L. (1986). Thought and language. Cambridge, MA: MIT Press.

Wallace, M. J. (1998). Action research for language teachers. Cambridge, England: Cambridge University Press.

Wenger, E. (1998). Communities of practice: Learning, meaning, and identity. Cambridge, U.K; New York, N.Y: Cambridge University Press.