Investigating Student Engagement in an Online Mathematics Course through Windows into Teaching and Learning

         Investigating Student Engagement in an Online Mathematics Course through Windows into Teaching and Learning

Teresa Petty
Associate Professor
Department of Middle, Secondary, and K-12 Education
University of North Carolina at Charlotte
Charlotte, NC 28223 USA
tmpetty@uncc.edu

Abiola A. Farinde
Graduate Research Assistant
Department of Middle, Secondary, and K-12 Education
University of North Carolina at Charlotte
Charlotte, NC 28223 USA
afarinde@uncc.edu

Abstract

The Windows into Teaching and Learning (WiTL) project is a method developed by researchers at the University of North Carolina at Charlotte that allows the facilitation of online clinical experiences for students during their content area methods course. WiTL was developed to address difficulties in securing meaningful clinical placements for interns during online summer coursework. WiTL encompasses both an asynchronous and a synchronous component. Through the use of dialogue, the purpose of WiTL is to engage interns with practicing classroom teachers and with each other as they consider various teaching methodologies and observe these methodologies in practice. The authors describe the WiTL process, its implementation, and ways in which the process has encouraged various levels of student engagement in an online mathematics methods course. The results indicate varying levels of student engagement and suggest that students are more engaged during the asynchronous portion of this study.

Keywords: online content methods courses, student engagement, distance learning, asynchronous online learning, synchronous online learning, online clinical experiences

Introduction

Researchers at one southeastern University were experiencing difficulties with clinical placements during online summer coursework. During a summer school session, the challenge to find either year-round schools or schools that have a later end of year so that methods students can be placed in a classroom to observe teaching practices is problematic. In the past, students were allowed to observe the behavior of adolescents in other settings such as summer camps and the Young Men's Christian Association (YMCA). These observations did not provide an ideal situation in which students could observe classroom teachers in their content area delivering instruction. In an effort to deliver meaningful learning experiences and bridge the gap between theory and practice, Windows into Teaching and Learning (WiTL) was conceptualized and implemented. WiTL offered online clinical observation experiences of practicing teachers in various grades and content areas.

The WiTL process utilized web-conferencing software supported by the University. Students were required to have access to a computer, the Internet, a headset, and a webcam to participate in WiTL activities. Students in methods courses observed the practices of teachers both synchronously and asynchronously. Following synchronous classroom observations, which were delivered via Wimba, methods students were given the opportunity to debrief with the practicing teachers, using a text chat feature in Wimba. Dialogue was exchanged regarding the pedagogical practices of the teachers and the rationale for using particular methodologies. Methods students also participated asynchronously by watching prerecorded videos of the practicing teachers. Following these viewings, they participated in an online threaded discussion, via NiceNet, with each other and the practicing teachers.

This research explored the engagement of university graduate students in an online middle grades/secondary mathematics methods course during summer clinical experiences. Student–student interactions (Moore, 1989) were closely examined through an analysis of the asynchronous threaded discussions and synchronous text chat communications using the engagement framework devised by Perkins and Murphy (2006). The examination revealed various levels of engagement among students in both the synchronous and asynchronous platforms. The guiding research question for this study was: To what extent do students in an online mathematics methods course engage in meaningful discourse and collaboration that encourages them to critically examine teacher pedagogy?

Literature Review

Teacher education programs, through the use of technology, offer online courses that seek to enhance student learning (Nandi, Hamilton, Harland, & Warburton, 2011). Of the many technology-based modes of instruction used in online teacher education courses, online discussion forums are often specifically used as a vehicle for the continual discussion of class topics and concepts. In addition, this learning environment, whether synchronous or asynchronous, facilitates the dissemination and acquisition of knowledge and enables student interaction and collaboration (Levine, 2007). Transforming the traditional learning context, these pedagogical tools allow greater access to classroom peers as well as a fluid exchange of content and resources with the intent of improving learning outcomes.

Although online teacher education courses are an innovative and unique pedagogical approach, their effectiveness is often questioned because student interaction in online discussion forums does not necessarily mean that students are actively engaged in the learning process (McLoughlin & Mynard, 2009; Robinson & Hullinger, 2008). In fact, solely fulfilling the specified participation and time requirements of an online course suggests that students are merely "doing time" rather than "doing education" (Zyngier, 2007). In doing time, students fail to go beyond the content, do not bridge theory and practice, and avoid authentic student engagement. Considering that deep learners are often more engaged than surface learners (Hockings, Cooke, Yamashita, McGinty, & Bowl, 2008) and that pre-service teachers' pedagogical training will influence the lives of their future students, engagement is a poignant topic for online teacher education courses.

When reviewing the literature, we found that at the collegiate level the definitions and measurement of online student-to-student engagement were limited. However, the literature did show that engagement is seen through numerous lenses (Zepke & Leach, 2010): student motivation (Schuetz, 2008), students' social and economic background (Pascarella & Terenzini, 2005), institutional support (Kuh et al., 2005), peer interaction (Moran & Gonyea, 2003), teacher behavior (Bryson & Hand, 2007), etc. Although engagement is often difficult to define and measure in online courses, its absence and presence is identifiable. Due to the study's focus on student-to-student engagement, for the purpose of this study engagement is defined by Kuh (2003) as the efforts of the student to study a subject, practice, obtain feedback, analyze, and solve problems. This definition illustrates that engagement cannot be captured through a snapshot but rather has an interpersonal component. Engagement is best observed through interactions with others and through a development of complex ideas. For instance, echoing this definition of engagement, Guthrie and Anderson (1999) state, "social interaction patterns in the classroom can amplify or constrict students' ... attainment of deep conceptual knowledge" (p. 20). Furthermore, within the confines of face-to-face (F2F) schooling, engagement is seen as a multidimensional concept: behavioral, cognitive, and emotional (Fredricks, Blumenfeld, & Paris, 2004). Behavioral engagement measures students' ability to follow school rules, their level of involvement in learning, and participation in extracurricular activity; cognitive engagement is students' investment in and active participation in learning as they move beyond required coursework; and emotional engagement is the positive and negative feelings students hold for school. Aligning – yet moving beyond – the aforementioned definitions, engagement in this study is measured in the context of students' interactions in an online mathematics methods course. Engagement can be identified in this particular learning environment as participants meaningfully contribute to online discussions. Within the asynchronous and synchronous forums, students commented, asked for clarification, posed questions, made inferences, assessed the learning environment, and collectively strategized about how to best bridge content and pedagogy. Their interactions with each other are the focal point of analysis and determine whether student-to-student engagement was present within the online discussions.

One of the challenges of online discussion forums is determining whether quality online engagement is actually occurring or whether students are passively interacting with their peers. In an era where more and more students are learning via online coursework (Murray, Pérez, Geist, & Hedrick, 2012), it is imperative that researchers determine if online learning can equally promote student engagement in comparison to a F2F learning environment. If student engagement is absent or minimized, then full cognitive development in a specific content area is unachievable. Because of this uncertainty regarding online engagement and due to the need to move beyond rote memorization of knowledge, an evaluation of online learning engagement levels must be performed to ensure positive learning outcomes for online students.

Synchronous (occurring in real time) and asynchronous (occurring over time) online discussion forums provide a space for open communication among students. Regardless of person or location, one's thoughts are transmitted to others, constructing new knowledge or further developing existing ideas. While both modes of communication serve a common purpose, asynchronous communication is often deemed more appropriate for facilitating active, meaningful engagement. Recognizing weakness in both communicative resources, Branon and Essex (2001) outline the disadvantages of both types of communication that may consequently hinder student engagement. They list the following restrictions when utilizing synchronous communication: (1) getting students online at the same time; (2) difficulty in moderating large-scale conversations; and (3) lack of reflection time. Similar to synchronous discussions, asynchronous communication equally places limitations on engagement due to the lack of immediate feedback, and the infrequency of students logging in to online discussion forums.

Students' level of engagement during synchronous and asynchronous communication is often challenged because students are not physically present in the same space, supposedly decreasing their opportunity to socially interact, collaborate, give and receive feedback, and render support (Tuckman, 2007). Although both modes of communication possess certain weaknesses, traditional F2F student communication does not guarantee meaningful student-to-student engagement. In fact, bodies in close proximity to one another, occupying a shared classroom, may promote interaction but may fail to facilitate active, engaged learning. Unlike traditional F2F student communication, there are numerous synchronous and asynchronous modes of communication, which are used to promote engagement. The diverse asynchronous (e-mail, listserv, discussion boards, and web logs) and synchronous (chat, instant messaging, and audio and web-based conferencing) modes of communication have the potential to increase interaction and improve online learning environments (Repman, Zinskie, & Carlson, 2005). Through the implementation of effective pedagogical learning principles delivered via online communication, a community of engaged, communicative learners may emerge (Peck, 2012).

As mentioned, although constraints are present in both communication tools, Schellens, Van Keer, and Valcke (2005) affirm that asynchronous discussion boards foster active engagement through a community of learners who teach one another and provide constructive feedback while sharing and gaining information. Im and Lee (2003) advocate the use of asynchronous over synchronous in generating a more effective learning environment. PytlikZillig et al. (2011) also compare synchronous F2F discussions and asynchronous computer-mediated discussions, asserting that computer-mediated discussions produce more effective engagement than F2F discussions.

Moreover, the flexibility and convenience of time, which allows for greater reflection and more collaboration among students, is often cited as a primary reason for why asynchronous discussion forums are a preferred learning resource (Ajayi, 2009; Nandi, Hamilton, & Harland, 2012). While extended time does not necessarily equate to student engagement, the opportunity to engage with others is greater, increasing the likelihood that such engagement may occur. The presence of this engagement, though logical, requires further empirical support. The need for more research on engagement within online forums (Su, Bonk, Magjuka, Liu, & Lee, 2005) reiterates the persistent question of whether students are actively engaged or passively participating. In seeking insight in this matter, we begin with the following research problem: Are students fully engaged in online synchronous and asynchronous courses?

Adding to the literature, this exploratory study examines synchronous and asynchronous student–student interactions (Moore, 1989) in an online middle grades and secondary mathematics methods course for graduate education students. Students' levels of engagement were coded using Perkins and Murphy's (2006) engagement framework. Perkins and Murphy's rubric measures individual engagement in critical thinking in an online asynchronous discussion. While this model was previously used in online asynchronous discussions, answering Perkins and Murphy's call for further research, this study sought additional empirical data by applying their model to asynchronous and synchronous discussions. Perkins and Murphy's model was deemed appropriate and applicable to asynchronous and synchronous modes of communication because different levels of engagement, which were illustrated by students' varying levels of critical-thinking questions and comments, were present in both discussion forums. Identification and measurement of student engagement in critical thinking in both settings revealed that regardless of the online medium, the implementation of sound, student-centered pedagogical practices will foster student engagement. Table 1 provides a detailed classification of the four engagement categories.

Table 1. Perkins and Murphy's (2006) model for identifying engagement in critical thinking

Clarification:
All aspects of stating, clarifying, describing (but not explaining), or defining the issue being discussed.Analyzes, negotiates, or discusses the meaning of the issue.
Identifies one or more underlying assumptions in a statement in the discussion.Identifies relationships among the statements or assumptions.Defines or criticizes the definition of relevant terms. Assessment:
Evaluating some aspect of the debate; making judgments on a situation, proposing evidence for an argument or for links with other issues.Provides or asks for reasons that proffered evidence is valid.Provides or asks for reasons that proffered evidence is relevant.Specifies assessment criteria, such as the credibility of the source. Makes a value judgment on the assessment criteria or a situation or topic.Gives evidence for choice of assessment criteria.Inference:
Showing connections among ideas; drawing appropriate conclusions by deduction or induction, generalizing, explaining (but not describing), and hypothesizing.Deduces relationships among ideas.Strategies:
Proposing, discussing, or evaluating possible actions.Predicts outcomes of proposed actions.Method

Project Description and Participants

The primary goal of WiTL was to provide a meaningful summer clinical experience for students enrolled in an online mathematics methods course. Data were collected from all students (n = 22) enrolled in the online summer mathematics methods course at the researchers' University. Researchers utilized a middle school and a high school based on faculty connections in working within the school and community. The three teachers of middle grades that were invited to participate in this project were identified by their principal as exemplary teachers. All three middle grade levels were represented: one teacher was a sixth-grade mathematics teacher, one a seventh-grade mathematics teacher, and one an eighth-grade mathematics teacher. The three high school teachers that were asked to join were also acknowledged by their principal as exemplary teachers. The three teachers collectively taught Algebra I, Algebra II, Geometry, Pre-Calculus, and Advanced Placement Calculus. Prior to the beginning of the summer methods course, each of the practicing teachers selected two exceptional lessons. These lessons were then videotaped during regular classroom instruction using a laptop, wireless headset, webcam, and TechSmith's Camtasia, a software application used for screen video capture.

The methods students viewed these 12 asynchronous videos (two per teacher) and then participated in an online dialogue regarding the practices they observed. These threaded discussions, facilitated through NiceNet, occurred over two weeks and allowed methods students to engage in conversation with the six practicing mathematics teachers and their peers. The conversations (examples provided in data analysis section) included dialogue between students regarding instructional methods and pedagogical thinking concerning the teaching and learning of mathematics for Grades 6 to 12.

The three middle school and three high school mathematics teachers were also asked to allow online mathematics methods students to participate in a live teaching observation. These synchronous teaching observations were facilitated using a computer connected to the Internet, a webcam, a wireless headset, and Saba Centra (the University-supported web conferencing software). These items were chosen because all online cohort students utilize these technological tools throughout their program coursework. Methods students logged on to Centra at the indicated class time to observe the teachers. They viewed six separate teaching episodes and participated in a text chat dialogue with their classmates during the observations. This viewing and interaction allowed students to ask one another questions regarding the classroom environment, classroom management, instructional methods, and pedagogical thinking.

Following the teaching observation, the classroom teacher joined the methods students in Centra to take comments about their teaching and answer questions regarding instructional decision making. Examples are provided in the Data Analysis section.

WiTL was a multifaceted project, and therefore, generated several data sources. These included individual interviews with each classroom mathematics teacher, focus group interviews with methods students at the conclusion of the semester, copies of the asynchronous threaded discussion forum, text chat logs of methods students conversations during the synchronous teaching observations, and archives of both the synchronous and asynchronous sessions. The data essential to this present study are the asynchronous threaded discussions, facilitated through NiceNet, and the text chat logs, facilitated during the synchronous teaching observations via Centra. The following research question guided the analysis of data:

To what extent do students in an online mathematics methods course engage in meaningful discourse and collaboration that encourage them to critically examine teacher pedagogy?

Data Analysis

The threaded discussions were analyzed to determine various levels of student-to-student engagement during online clinical experiences. Researchers used content analysis to quantify levels of engagement based on students' questions and comments in the threaded discussions. Students' levels of engagement with each other were coded using Perkins and Murphy's (2006) engagement framework presented in Table 1. Various levels of analysis occurred. Initially, the researchers evaluated the data independently, determining which level of engagement was represented in the dialogue. Next, the researchers compared their analyses and determined inconsistencies. Finally, inconsistencies were discussed until final consensus was reached. This method of analysis allowed the researchers to establish inter-rater reliability (Neuendorf, 2002). Frequencies for each category were then tabulated.

The text chat logs, generated during the synchronous teaching observations, were analyzed in a similar manner using content analysis. Again, students' levels of engagement with each other were coded using Perkins and Murphy's (2006) engagement framework (see Table 1). Researchers conducted an independent analysis of the data to decide which level of engagement was represented in the various comments and questions stated during text chat sessions. Researchers then compared their analyses to determine inconsistencies. Once more, inconsistencies were debated until consensus was reached, allowing for the establishment of intercoder reliability (Neuendorf, 2002). Frequencies for each category were then recorded.

Results

After the threaded discussions and text chat logs were analyzed, relative frequencies were tabulated. These data are presented in Table 2.

The data, although not conclusive, indicate varying levels of student engagement across both synchronous and asynchronous platforms. This distribution of student outputs across the engagement categories yields interesting results. The asynchronous forum postings show a higher percentage of Strategy engagement (15.2% vs. 7.9%) as well as a higher percentage of Clarification engagement (61% vs. 45.7%). The synchronous forum postings indicate something different. The categories of Assessment (35.3% vs. 17.5 %) and Inference (11.1% vs. 6.3%) indicate higher levels of engagement for the synchronous platform.

Table 2. Frequency of student output – synchronous vs. asynchronous

Asynchronous
(Threaded Discussion) It is important to note that the threaded discussion forum was available to students and classroom teachers for two weeks, while the text chat logs were offered during six synchronous teaching observations with durations of 75 to 90 minutes each. All four categories of engagement were observed in both the threaded discussions and text chat logs. Students in both settings engaged more at the level of clarification. This was apparent as students asked questions and made comments regarding the classroom context (i.e., configuration of the desks/classroom, decoration of the classroom, technology available to students, academic levels of students, etc.). As the level of engagement increased on Perkins and Murphy's (2006) engagement framework, the relative frequency of students' comments and questions decreased with the exception of the Strategies category. This could be warranted due to the time required for students to reflect on teachers' actions and offer possible justifications. All four categories were observed, but as noted above, engagement cannot be solely measured by frequency (doing time). A critical examination of the actual content presented by students during the threaded discussions and text chat dialogues must also be considered when determining students' engagement level.

The data presents examples of authentic student engagement. For example, the following are distinct comments and statements posted by one student and responded to by another student during the asynchronous online discussion forum (NiceNet). In this exchange students share questions they still have about slope and direct variation and suggestions for teaching systems of equations. This exchange reinforces the notion that students can learn important teaching strategies from one another when they have the opportunity to discuss their ideas. The level of engagement based on Perkins and Murphy's (2006) framework is indicated [in brackets, and rendered in bold].

Student 1: "I really like how you [the teacher] make the association between slope and direct variation because that is a topic that my ninth-grade Algebra I students have a hard time with. Do you have any other suggestions on how to make it clearer? [Strategies] Also why did they change the variable to "k" instead of leaving it as "m"? [Clarification] I am a math major (not an education major) and I still am not sure why the variables in direct variation and slope intercept changed. I know it is a little late in the year for this stuff and you may have done this before, but there is a cool activity to do with solving systems. You break the students in groups and have them do an advertisement for a particular method. I make my students show examples, write a definition, make a logo, a slogan and a valid argument as to why their method is better than all the others. I also make them present it in front of the class. They usually have fun with it. [Strategies]"

Student 2: "I am also a math major and have never understood why the variable changed. I look forward to the teacher's answer! [Clarification] Also, thanks for sharing that method! Systems of equations was the topic I struggled the most with teaching this year and am concentrating on to make it better for next year. I love your project idea and am definitely going to implement that into my classroom! [Assessment]"

The exchange below is also taken from the NiceNet threaded discussion. It exemplifies quality engagement during student-to-student interaction. The students really push each other to think about the topic of fractions and consider various situations in which fractions are used. Both students are making connections, providing examples, going beyond the content, and actively contributing to the discussion.

Student 3: "When in real life do we ever really use fractions? I can think of obvious ones: slices of pizza, talking about discounts and sales tax, but other than those, I'm not sure I really know if we DO use fractions. [Assessment]"

Student 4: "I would argue that we use fractions every time we have to use division. If you look at a class and want to know what percentage of the students did their homework, you are going to have to collect the raw data first. If 16 out of 20 students did their homework, that is a fraction, 16/20. After you know that fraction, you are able divide and determine what that number is as a decimal and percentage. I agree that numbers are not usually left as fractions, and that fractions do not always look as polished as decimals do, but they are something that is used in many people's daily lives. [Assessment]"

Student 3: "Thank you for pushing my thinking. However, if we are using a fraction to solve for a percent on a test, wouldn't we convert it to a decimal to understand it better? Maybe it is the way I think and not the way the world works, but I understand 80% much better than I understand 16/20 even though they represent the same amount. Also, will we ever have to multiply, divide, add, or subtract fractions in real life? I absolutely agree that students need to be able to convert fractions to decimals to percents and vice versa, but will it hurt them to use a calculator to always compute with fractions (as in adding, subtracting, multiplying, and dividing)? I am not sure I know the answer to that question ... [Assessment]"

The following are individual comments made by students to students during the synchronous teaching observations via the text chat feature in Centra. In this exchange, students are discussing the note taking methods used by the teacher they are observing. This exchange reveals the ideas that the students have regarding this technique. The level of engagement based on Perkins and Murphy's (2006) framework is indicated.

Student 1: "How structured do you think note taking should be? Right now, they're basically writing exactly what he dictates. [Clarification]"

Student 2: "I think it will depend on my students ... may have to feel it out ... [Assessment]"

Student 3: "I think that note taking is a skill that has to be explicitly taught. [Assessment]"

Student 4: "I agree that it has to be taught, but you can't learn if you always are just copying something ... [Assessment] Maybe this is a good method for younger students who are still in the beginning stages, though. [Inference]"

The exchange below is also taken from the text chat log. During this exchange, students are observing a teacher-centered classroom. The teacher is initiating traditional instructional methods. The students are offering suggestions for getting the middle school students motivated as many of them are visibly disengaged.

Student 1: "Maybe some cooperative learning activities and more game type learning would motivate them a little more. Make the students think they're playing instead of learning. [Strategies]"

Student 2: "They will think it is a reward for passing the test. [Strategies]"

Student 3: "The teacher should consider maybe not teaching new information but readdressing what they did not get. [Strategies]"

The comments/questions presented above demonstrate the various levels of engagement that students exhibited as they participated in both the threaded discussions and the text chat dialogues. Students discuss a variety of issues that they see as they observe teaching practices via synchronous and asynchronous observations.

Limitations

As with most research studies, this study does have its limitations. The most apparent limitation is the small sample size, making the results difficult to generalize. The study was conducted over a short period of time – one semester during the summer. In the course of this study, there were technical difficulties. During two of the synchronous teaching observations, the teachers had difficulty with their wireless headsets, which resulted in periods of no sound for the online methods students who were viewing the teaching observations. This issue was quickly resolved. Another possible limitation relates to the asynchronous threaded discussion. Since this is a monitored forum, students may have felt more inclined to participate and to provide thoughtful responses.

The use of the Perkins and Murphy (2006) framework could also be viewed as a limitation. It was first developed for use with the asynchronous platform. This study took it a step further and utilized it with both asynchronous and synchronous platforms.

Conclusion

In this study, the researchers explored the guiding research question: To what extent do students in an online mathematics methods course engage in meaningful discourse and collaboration that encourage them to critically examine teacher pedagogy? It was determined that students engage in all levels of engagement as defined by Perkins and Murphy (2006) in both asynchronous and synchronous platforms. While asynchronous forum postings show a higher percentage of Strategy engagement (15.2% vs. 7.9%) as well as a higher percentage of Clarification engagement (61% vs. 45.7%), the synchronous forum postings indicate something different. The categories of Assessment engagement (35.3% vs. 17.5 %) and Inference engagement (11.1% vs. 6.3%) presented higher levels of engagement for the synchronous platform. The settings of the individual platforms would certainly encourage this. In the asynchronous environment, students would have more time to reflect and therefore offer strategies. Clarification engagement may be more present in the asynchronous platform because students in the synchronous platform were afforded opportunities to get clarification to these questions during the synchronous settings that were not readily available in the asynchronous setting. This notable constraint does differ from the literature, which asserts that asynchronous is preferable. The results of the WiTL study suggest that the asynchronous and synchronous platforms are emphasizing different types of engagement. Future research studies are needed to determine which method is more beneficial, or if one method is better at engaging students in an online environment.

WiTL proved to be a beneficial experience for methods students. It offered outreach to students across the state through an innovative approach to clinical experiences. WiTL provided the online methods students not only the opportunity to view sound teaching practices, it also allowed them the chance to engage with practicing mathematics teachers to confirm their understandings of methodologies and pedagogical decision making. During a typical clinical observation experience, a student would be placed with one teacher and one classroom. In this online mathematics methods course, students were placed in different classrooms across a variety of schools. WiTL was a unique experience in that it allowed online methods students to observe six different teachers in two different schools and school systems, exposing them to an assortment of teaching styles in a variety of classroom environments. Since WiTL allowed the online methods students to observe the same teachers, an exchange of thoughts and ideas related to the instructional strategies they observed was also facilitated. The WiTL process was equally valuable to the practicing mathematics teachers as it allowed them the opportunity to critically reflect on their teaching practices as they considered questions that the methods students posed. Through WiTL, methods students were able to utilize technology to engage in online clinical experiences at various levels with their peers.

References

Ajayi, L. (2009). An exploration of pre-service teachers' perceptions of learning to teach while using asynchronous discussion board. Educational Technology & Society, 12(2), 86-100. Retrieved from http://www.ifets.info/journals/12_2/7.pdf

Branon, R. F., & Essex, C. (2001). Synchronous and asynchronous communication tools in distance education. TechTrends, 45(1), 36-42. doi:10.1007/BF02763377

Bryson, C., & Hand, L. (2007). The role of engagement in inspiring teaching and learning. Innovations in Education and Teaching International, 44(4), 349-362. doi:10.1080/14703290701602748

Fredricks, J. A., Blumenfeld, P. C., & Paris, A. H. (2004). School engagement: Potential of the concept, state of the evidence. Review of Educational Research. 74(1), 59-109. doi:10.3102/00346543074001059

Guthrie, J. T., & Anderson, E. (1999). Engagement in reading: Processes of motivated, strategic, knowledgeable, social readers. In J. T. Guthrie & D. E. Alvermann (Eds.), Engaged reading: Processes, practices, and policy implications (pp. 17-45). New York, NY: Teachers College Press.

Hockings, C., Cooke, S., Yamashita, H., McGinty, S., & Bowl, M. (2008). Switched off? A study of disengagement among computing students at two universities. Research Papers in Education, 23(2), 191-201. doi:10.1080/02671520802048729

Im, Y., & Lee, O. (2003). Pedagogical implications of online discussion for preservice teacher training. Journal of Research on Technology in Education, 36(2), 155-170.

Kuh, G. D. (2003). What we're learning about student engagement from NSSE: Benchmarks for effective educational practices. Change: The Magazine of Higher Learning, 35(2), 24-32. doi:10.1080/00091380309604090

Kuh. G. D., Kinzie, J., Schuh, J. H., & Whitt, E. J., & Associates. (2005). Student success in college: Creating conditions that matter. San Francisco, CA: Jossey-Bass.

Levine, S. (2007). The online discussion board. New Directions for Adult and Continuing Education, 113, 67-74. doi:10.1002/ace.248

McLoughlin, D., & Mynard, J. (2009). An analysis of higher order thinking in online discussions. Innovations in Education and Teaching International, 46(2), 147-160. doi:10.1080/14703290902843778

Moore, M. G. (1989). Three types of interaction. American Journal of Distance Education, 3(2), 1-6. doi:10.1080/08923648909526659

Moran, E. T., & Gonyea, T. (2003). The influence of academically-focused peer interaction on college students' development. Available from ERIC database. (ED478773)

Murray, M., Pérez, J., Geist, D., & Hedrick, A. (2012). Student interaction with online course content: Build it and they might come. Journal of Information Technology Education: Research, 11, 125-140. Retrieved from http://www.jite.org/documents/Vol11/JITEv11p125-140Murray1095.pdf

Nandi, D., Hamilton, M., & Harland, J. (2012). Evaluating the quality of interaction in asynchronous discussion forums in fully online courses. Distance Education, 33(1), 5-30. doi:10.1080/01587919.2012.667957

Nandi, D., Hamilton, M., Harland, J., & Warburton, G. (2011). How active are students in online discussion forums. In J. Hamer & M. de Raadt (Eds.), Proceedings of the Thirteenth Australasian Computing Education Conference: Vol. 114 Conferences in Research and Practice in Information Technology (pp. 125-134). Darlinghurst, Australia: Australian Computer Society. Retrieved from http://www.crpit.com/confpapers/CRPITV114Nandi.pdf

Neuendorf, K. A. (2002). The content analysis guidebook. Thousand Oaks, CA: Sage.

Pascarella, E. T., & Terenzini, P. T. (2005). How college affects students: A third decade of research. San Francisco, CA: Jossey-Bass.

Peck, J. J. (2012). Keeping it social: Engaging students online and in class. Asian Social Science, 8(14), 81-90. doi:10.5539/ass.v8n14p81

Perkins, C., & Murphy, E. (2006). Identifying and measuring individual engagement in critical thinking in online discussions: An exploratory case study. Educational Technology & Society, 9(1), 298-307. Retrieved from http://www.ifets.info/journals/9_1/24.pdf

PytlikZillig, L. M., Horn, C. A., Bruning, R., Bell, S., Liu, X., Siwatu, K. O., ... Carlson, D. (2011). Face-to-face versus computer-mediated discussion of teaching cases: Impacts on preservice teachers' engagement, critical analyses, and self-efficacy. Contemporary Educational Psychology, 36(4), 302-312. doi:10.1016/j.cedpsych.2011.07.002

Repman, J., Zinskie, C., & Carlson, R. D. (2005). Effective use of CMC tools in interactive online learning. Computers in the Schools, 22(1-2), 57-69. doi:10.1300/J025v22n01_06

Robinson, C. C., & Hullinger, H. (2008). New benchmarks in higher education: Student engagement in online learning. Journal of Education for Business, 84(2), 101-109. doi:10.3200/JOEB.84.2.101-109

Schellens, T., Van Keer, H., & Valcke, M. (2005). The impact of role assignment on knowledge construction in asynchronous discussion groups: A multilevel analysis. Small Group Research, 36(6), 704–745. doi:10.1177/1046496405281771

Schuetz, P. (2008). A theory-driven model of community college student engagement. Community College Journal of Research and Practice, 32(4-6), 305-324. doi:10.1080/10668920701884349

Su, B., Bonk, C. J., Magjuka, R. J., Liu, X., & Lee, S.-H. (2005). The importance of interaction in web-based education: A program-level case study of online MBA courses. Journal of Interactive Online Learning, 4(1), 1-19. Retrieved from http://www.ncolr.org/jiol/issues/pdf/4.1.1.pdf

Tuckman, B. W. (2007). The effect of motivational scaffolding on procrastinators' distance learning outcomes. Computers & Education, 49(2), 414-422. doi:10.1016/j.compedu.2005.10.002

Zepke, N., & Leach, L. (2010). Improving student engagement: Ten proposals for action. Active Learning in Higher Education, 11(3), 167-77. doi:10.1177/1469787410379680

Zyngier, D. (2007). (Re)conceptualising student engagement: Doing education not doing time. Teaching and Teacher Education, 24(7), 1765-1776. doi:10.1016/j.tate.2007.09.004

View the original article here

Patterns of Engagement in Connectivist MOOCs

         Patterns of Engagement in Connectivist MOOCs

Colin Milligan
Research Fellow
Caledonian Academy
Glasgow Caledonian University
Glasgow G4 0BA UK
colin.milligan@gcu.ac.uk

Allison Littlejohn
Professor and Director
Caledonian Academy
Glasgow Caledonian University
Glasgow G4 0BA UK
allison.littlejohn@gcu.ac.uk

Anoush Margaryan
Senior Lecturer
Caledonian Academy
Glasgow Caledonian University
Glasgow G4 0BA UK
anoush.margaryan@gcu.ac.uk

Abstract

Connectivist massive open online courses (cMOOCs) represent an important new pedagogical approach ideally suited to the network age. However, little is known about how the learning experience afforded by cMOOCs is suited to learners with different skills, motivations, and dispositions. In this study, semi-structured interviews were conducted with 29 participants on the Change11 cMOOC. These accounts were analyzed to determine patterns of engagement and factors affecting engagement in the course. Three distinct types of engagement were recognized – active participation, passive participation, and lurking. In addition, a number of key factors that mediated engagement were identified including confidence, prior experience, and motivation. This study adds to the overall understanding of learning in cMOOCs and provides additional empirical data to a nascent research field. The findings provide an insight into how the learning experience afforded by cMOOCs suits the diverse range of learners that may coexist within a cMOOC. These insights can be used by designers of future cMOOCs to tailor the learning experience to suit the diverse range of learners that may choose to learn in this way.

Keywords: massive open online course (MOOC), connectivist massive open online course (cMOOC), connectivism, lurking, networks, active participation, passive participation

Introduction

Pedagogical models of learning online have been extensively theorized, particularly with respect to the interrelationship of technology and pedagogy (Anderson & Dron, 2010; Garrison, 1997; Kanuka & Anderson, 1999). As we approach near ubiquity of networked connections between people, content, and tools (reflecting the "networked society" described by Castells, 1996), researchers such as George Siemens have proposed new pedagogical approaches based on the principles of connectivism (Siemens, 2005), through which learning is viewed as residing in the connections that exist between people and digital artifacts within this ubiquitous network. One example of connectivist pedagogy in action is the massive open online course (MOOC) format pioneered by Siemens along with his colleague Stephen Downes, first in the Connectivism and Connected Knowledge 2008 (CCK08) course (Downes, 2008) and thereafter in many subsequent courses including the Change11 course that forms the basis of this study. These MOOCs, known as connectivist or cMOOCs, focus on knowledge creation and generation rather than "knowledge duplication" (Siemens, 2012, para. 3). In cMOOCs, the learners take a greater role in shaping their learning experiences than in traditional online courses, while facilitators focus on fostering a space for learning connections to occur. While cMOOCs can empower learners to take control of their learning, there remains a question about how the learning experience afforded by these cMOOCs is suited to learners with different skills, motivations, and dispositions. As cMOOCs are a relatively new phenomenon, there are few studies that have explored these issues (e.g., Kop & Fournier, 2010; Mackness, Mak, & Williams, 2010). The overall aim of this study is to address this lack of empirical data.

This study focuses on two research questions:

What patterns of engagement exist within the Change11 cMOOC course?What principal factors mediate this engagement?By gaining a deeper insight into the patterns of engagement in cMOOC courses, this study seeks to provide an insight into how future cMOOCs can be designed to better support the learning needs and expectations of the wide range of learners that coexist within cMOOCs. The paper begins with a review of relevant literature, focusing on existing research examining the learning experience in cMOOCs. Next, the Change11 course context is described, followed by a description of the methodology adopted in this study and the sample studied. The study findings are then presented and discussed. Finally, recognizing the limitations of this study, the implications of these findings for research and practice are discussed, particularly in relation to the design of future cMOOC learning experiences.

Literature Review

Connectivist MOOCs emerged initially as an instantiation of the pedagogic principles of connectivism developed by Siemens and Downes, and their first MOOC, CCK08, naturally explored the topic of connectivism as it attracted participation from learning researchers and practitioners who had been following the evolution of these ideas. Subsequent large scale cMOOCs, such as Personal Learning Environments Networks and Knowledge (PLENK 2010) and Connectivism and Connected Knowledge 2011 (CCK11), have continued in the same vein, exploring similar topics and attracting participants eager to experience the cMOOC format as well as the course content itself. This has led to the emergence of a rather unusual research base where a small amount of empirical research, published in niche journals and peer reviewed conferences, is supplemented by a large body of more anecdotal and reflective research published outside the traditional peer-reviewed journal system. To date, the key empirical research has been carried out by three groups of researchers: Fini (2009), Mackness et al. (2010), and Kop and her colleagues (Kop, 2011; Kop & Fournier, 2010; Kop, Fournier, & Mak, 2011), exploring the cMOOCs listed above.

Software tools for discovery, connection, and co-creation are a fundamental component of cMOOCs, together constituting the platform through which the course is delivered. In his study, Fini (2009) focused on the technological dimension of CCK08, exploring participants' perceptions of the course toolset as well as their views on the course. The research showed that one course component, The Daily newsletter, was valued by participants who used it as a tool to filter and organize their participation in the course, but that other tools, such as the discussion forums provided by Moodle, were viewed less positively. In addition, Fini's research highlighted digital literacy and English language proficiency as key skills needed to participate effectively in the course. The study also provided evidence of different behavior patterns being exhibited by course participants based on their personal objectives, backgrounds, and levels of engagement. As the first cMOOC, CCK08 can be seen as a prototype; and it is interesting to note that subsequent cMOOCs designed by Siemens and Downes adopted different toolsets, specifically ending the use of Moodle as a locus for discussion forums. This had the effect of moving discussion and interaction to blog posts and other spaces controlled by the learners, following the concept of the personal learning environment introduced by Wilson et al. (2007).

Mackness et al. (2010) also studied the CCK08 course, focusing on the four key characteristics of connectivist online courses suggested by Downes (2009): autonomy, diversity, openness, and connectedness/interactivity. Mackness et al.'s study, conducted by survey and e-mail interview, identified inherent tensions between these characteristics. The presence of these tensions raises the question of whether all four characteristics can be accommodated within the concept of a course in the traditionally understood sense, where the term creates expectations of structure, moderation, and support.

Research by Kop and associated researchers focused on the PLENK2010 cMOOC and carried on to examine the CCK11 cMOOC. This research focuses on the learning experience, and in particular on self-directed learning, asking whether cMOOCs facilitate the occurrence of self-directed learning. Kop and Fournier (2010) used quantitative (surveys) and qualitative (ethnography) data, supplemented by social network analysis to develop an understanding of autonomous learning within the PLENK2010 cMOOC. The research used Bouchard's (2009) four-dimensional model of learner control (conative, algorithmic, semiotic, and economic) to explore how motivation and confidence, the structure of learning, the delivery environment, and the perceived value of learning influenced learning strategies used by participants within the course. The study highlights additional factors necessary for learning in cMOOC type environments, principally the critical literacies essential to efficiently evaluate the large quantities of information present in a cMOOC – including an open mindset, the ability to learn cooperatively with others, and heightened critical analysis skills. In another paper based on the same study, Kop (2011) focuses on activities typical of cMOOC (or network)-based learning: aggregation, relation, creation, and sharing. The active and collaborative nature of these activities, similar to other classifications, such as the 4C learning behaviors (consume, connect, create, and contribute) identified by Littlejohn, Milligan, and Margaryan (2011), emphasize the importance of learning as a participatory process (Sfard, 1998) in the context of networked informal learning. Kop (2011) argues that to learn effectively in connectivist environments, learners must have key critical literacies such as those outlined above to be able to engage and participate, and have the confidence and competence to use the tools that mediate the key learning interactions that occur.

The Kop (2011) study also highlights the different types of learning approaches observed in the PLENK2010 cMOOC, recognizing the large number of lurkers (Rovai, 2000) who were present in these courses. In a third paper based on PLENK2010 but also incorporating data from the CCK11 cMOOC, Kop et al. (2011) return to the question of the underlying technology and explore whether specific learning environment designs can support effective learning in cMOOCs, arguing that structures must, at least to some extent, be emergent, and owned by the participants rather than imposed by the facilitators.
These studies have begun to provide a base of empirical data representing the learning that occurs in cMOOCs, but there is widespread recognition that more empirical research is urgently needed, particularly given the current widespread interest in the broad spectrum of MOOCs (Daniel, 2012). There is still a lack of data, drawn from too few courses, with a limited range of methodologies. This study seeks to build on these studies, examining a different course, the Change11 cMOOC, and using a different method (interview and survey instrument combined), to develop a greater understanding of the different patterns of engagement that coexist within cMOOC courses. Of particular interest is the nature of the networks these participants utilize to support their learning and the different factors that affect participation and engagement.

Course Context

The Change11 course was a large-scale cMOOC running from September 2011 to May 2012, organized and facilitated by George Siemens, Stephen Downes, and Dave Cormier. Over 35 weeks, participants were introduced to the work of a range of instructional design researchers and practitioners. Registration was open and course delivery was supported through a variety of technologies, principally a daily e-mail newsletter and online synchronous seminars delivered via the Elluminate platform. The course attracted more than 2,300 participants. An e-mail newsletter called The Daily communicated course announcements and content from presenters, while also aggregating blog posts and tweets from participants marked with the hashtag "#change11." This hashtag, along with self-organized spaces such as a Change11 group on Facebook and a multi-author blog associated with the course, provided a universal means of discovery of course related blog posts and content.

Method

The findings reported here represent one component of a larger study examining self-regulated learning behavior within the Change11 cMOOC. Participants for the study were recruited via an invitation and study description included as the first item in The Daily e-mail sent to everyone registered for the Change11 cMOOC during Week 17 of the course. Thirty-five individuals from a total of 2,300 registered learners agreed to participate. Study participants were initially invited to complete a short online survey. The survey used items adapted from a number of existing self-report self-regulated learning (SRL) instruments (Barnard, Lan, Paton, & Lai, 2009; Gijbels, Raemdonck, & Vervecken, 2010; Maclellan & Soden, 2006; Pintrich, Smith, García, & McKeachie, 1991; Schraw & Dennison, 1994; Toering, Elferink-Gemser, Jonker, van Heuvelen, & Visscher, 2012) to enable the researchers to derive SRL profiles for each participant. The survey also collected demographic information (country of residence, employer, and discipline/field) and asked respondents about their experience of previous MOOCs. From the initial sample of 35, 29 study participants were able to subsequently take part in a one-hour semi-structured interview via Skype, which explored various aspects of participation including motivation, goal-setting, and planning strategies, as well as exploring study participants' existing and emergent learning networks, their use of tools to support their learning, and their perceptions of their own participation in the course. Interviews were transcribed and stripped of identifying information. Transcripts were analyzed and a combination of predefined and emergent codes used to categorize the data. Ethical standards for the study were adopted in accordance with local regulations: all participants were provided with information about the study and how their data would be used, and were assured confidentiality and anonymity in any published work. Participants were asked to formally indicate their consent and were free to withdraw at any point.

Sample

Twenty-nine course participants out of 2,300 registrants on the course were interviewed in this study. Although a small number as a proportion of the total registrants, the sample is robust and appropriate for a qualitative study where a minimum sample of 12 is considered acceptable (Guest, Bunce, & Johnson, 2006). Everyone registered for the course received the e-mail that included the invitation to participate, so all those registrants who were active at Week 17 (midway through the course) would see the invitation. One limitation of the sampling procedure used is that anyone who had stopped actively following the course by this point would not have participated in the study. This limitation will be discussed in the results and discussion sections. The overarching topic of the cMOOC was instructional design, and all participants were either learning professionals (24, employed as lecturers, teachers, or instructional designers), or graduate students (five). Twelve of 24 learning professional participants were from the higher education sector, while 11 were drawn from the K-12 or community college sector. One participant was employed to support learning in a workplace outside the education sector. Eighteen of 29 participants were female, with 11 being male. Twenty of 29 participants had studied in previous cMOOCs.

Results

This study addressed two research questions: "What patterns of engagement exist within the Change11 cMOOC course?" and "What principal factors mediate this engagement?" These questions are now addressed in turn.

Patterns of Engagement

The semi-structured interview questions explored the topic of engagement in the course in a number of ways. Questions regarding motivation to do the course were complemented by questions exploring participation behavior. Finally, some questions were designed to probe the makeup of each respondent's personal learning network (their primary networks). Analyzing the responses to these questions allowed the authors to detect a number of different patterns of engagement with the course that are presented in this section.

Three distinct levels of engagement exhibited by the participants were identified in this study: Active participants, lurkers, and passive participants. Within these three levels, one key internal difference was observed: the location of the primary network for each individual, which might be internal or external to the course. Table 1 shows the number of participants in each category.

Table 1. Engagement and primary networks for the Change11 MOOC

Primary Network for this CourseThe different categories are described in turn below.

Active participants. The first group identified by this analysis (12 of 29) was a group described as active participants. These participants have adapted well to the connectivist pedagogy of cMOOCs, maintaining active blogs and Twitter accounts, actively and regularly discussing the course. All these active participants had formed wholly or primarily internal networks, connecting with other learners through Twitter and blogs. The live Elluminate sessions represented a key opportunity to make connections and widen their network. One participant, from South America, described the buzz of the chat sessions accompanying the live Elluminate sessions as follows: "You can read the comments of people who are participating from different places and they give links to things that they are doing or they think while you hear what is happening" (Participant 20). Not all active participants attended the live sessions (for reasons such as impractical scheduling in their time zone), but these sessions did seem to be an important focus for those who could attend.

Active participants were highly motivated to persist with the course and were able to overcome challenges in the course that might have proved a barrier to participation for others. For example one participant described how the live Elluminate sessions were sometimes confusing, and how she overcame this challenge:

"Yes I do, I found that when I was coming in cold to those speakers I was totally lost. So I had to come up with some short, sharp, effective strategies. First one being I'll check out the person on YouTube, I need to be able to get into the same space about what they're talking about, so at least I've got some sense of who they are and what they're going on about. So that was one strategy I used." (Participant 9)

Most active participants were energetic bloggers and Twitter users, using these tools as their main mechanism of communication. Active participants recognized that full participation entailed more than merely broadcasting ideas (creating tweets and blog posts) and had developed strategies to encourage connection with other participants through commenting on other blogs. One participant described how the majority of her contributions were made as responses to other posts: "I have no idea how scattered I am across this MOOC, I have no idea how many contributions I've made, 30? 50? I've got a lot of replies" (Participant 5). This participant went on to describe how she sought to encourage interaction: "So I usually end a reply on an open end. That's one of my, you know, I structure it that way" (Participant 5).

Aside from writing and commenting on blogs and following Twitter, active participants with primarily internal networks described other spaces that were particularly conducive to connection and collaboration. These spaces, established by the participants rather than facilitators, became the most vibrant community spaces for the course. One such space was Facebook, as described by one participant: "I found the Facebook group was where I really engaged with the course and teamed up with a few people there and it was good" (Participant 8). The same participant highlighted the value of some particularly active participants who could be inspirational to the rest of the cohort: "Oh there's some people who are everywhere you turn in the Change11 MOOC: there's this group of people who are inspirational, just phenomenal the way they just keep going and they know their way around it" (Participant 8).

While the majority (nine of 12) of active participants appeared to segregate their internal and external networks (effectively treating the Change11 as a closed course) a small number of participants (three of 12) seemed to have merged these networks to some extent – recognizing the artificiality of trying to separate different networks. One participant described how she actively sought to bring the two networks together when it was appropriate to do so:

"I try to think about taking what I find relevant in the [MOOC] and specifically advocating for it within [my external network] as well. So I do cross-pollinate like that. ... I hope that's helpful to people who normally wouldn't see this kind of stuff." (Participant 25)

The same participant summed up the value of being active in the course as follows, referring both to her motivation and to the development of her digital skills:

"I've gone further with the tools than I had in the past, so getting really familiar with Twitter, with Facebook, with LinkedIn, with Diigo for social bookmarking, for blogging, for setting up a blog ... and I remember when I crossed the threshold of having 1000 views ... yeahhh! It felt good to see the attraction, to have my work referred elsewhere." (Participant 25)

While a connectivist MOOC should cater to all types of participants, active participants represent the ideal learners for this type of course design: not just consuming content, but connecting with others, creating new content, and contributing these new resources back into the course for others to utilize. In fact, without a critical mass of active participants, a connectivist course would fail. Lurkers. The largest category of engagement identified in this analysis was of lurkers (13 of 29). These participants were actively following the course but did not actively engage with other learners within it. These participants were by no means disengaged with the course, or unhappy with their position. Instead, lurking was an active choice for them. One participant offered the view that "lurking is actually hugely beneficial" (Participant 18), before going on to describe how the course content was effectively new knowledge filtered by the course organizers and therefore had "more value than something I randomly come across on the Internet" (Participant 18).

As before, different types of lurking behavior were seen, based on the networks (external or internal) these lurkers engaged with. We saw three subcategories: (1) those who felt they did not engage with any network at all (four of 13); (2) those who engaged with networks external to the Change11 course but not with the internal networks (four of 13); and finally, (3) those who silently participated in internal networks (five of 13). Four of the participants did not engage with any network at all, and had a simple explanation for their behavior: that they were not interested in engaging with others to learn. One explained: "I guess I tend to be a loner and I've done more lurking and I'm quite happy lurking, I think it's an honourable profession" (Participant 21), while another simply stated: "I have not created any links with people, I have not said who I am and what I'm doing" (Participant 15). Even though this group had not engaged with others, the fact that they were still following the course after almost five months shows that the course format is compatible with their needs.

Four participants maintained a complex position whereby they were inactive within the course, but actively shared ideas from the course externally. As one participant reported:

"I'm going out to the MOOC and lurking and getting lots of great interesting ideas and I'm bringing these back to some of my home based networks, both ones within my institution but also ones within my network that I've built up professionally." (Participant 1)

A second participant made a similar point:

"I'm more or less like what do you call? A lurker and not very active ... I'm always invisible and the reason is that the way I've been using the MOOC is to put into things that I'm doing. Like I said, to be a network mentor." (Participant 17)

Both these participants had a clear understanding of why they wanted to participate in Change11. They wanted to apply the new knowledge they had gained to improve their own practice. Similarly, a third lurker in this group had shared her experience in the Change11 MOOC with colleagues at her own institution, while showing little engagement with the course community.

Finally, a group of five participants (all of whom self-identified as lurkers) silently participated in internal networks but did not contribute to the course in any way. Their behavior appears to be motivated by lack of confidence. For example, one participant provided the following explanation for her lurking:

"No, because I basically, I got caught up in my own learning and I didn't feel ... [it was worthwhile to contribute my] limited knowledge about what was being discussed. Beyond saying 'oh that resonates with me' well how many times have MOOC'ers said that! And I know it does resonate, but beyond that I couldn't add anything new." (Participant 18)

Others saw silent participation as a step to more full participation in future courses, acknowledging that in this course, they did not quite have the confidence to participate actively. As one participant remarked: "I did write a blog post but I conveniently wrote it somewhere no one would read [it] and I wrote it in [not English], so more like cathartic to me, rather than me putting it out in the open" (Participant 4). These silent lurkers saw a connection between level of participation and their success as learners, as exemplified by this quote from one participant: "I would have felt I accomplished more if I had personally networked and participated more" (Participant 33).

The lurkers category is somewhat complex as it includes a spectrum of participants from those who lacked the confidence to participate, to those who were so confident they didn't need to participate in the course in what they might regard as "the traditional manner." What links all these lurkers is that a cMOOC format works for them – they have the skills to leverage what they want from the course, on their terms.

Passive participants. The final group identified was the passive participant group. This category of four participants was united by their apparent frustration or dissatisfaction with the course. For one participant, the connectivist nature of the course just did not seem appropriate, as this extensive quote illustrates:

"I wrangled with the whole issue of connectivism, not the concept so much, as my ability to do that, to connect with others, and so I think I was looking inward a little bit and had some difficulty there as to whether or not I could succeed in that aspect as a learner, to be able to really make connections with other people on a deep level. I mean, sure, I can read other people's blogs and that's not a problem, and I comment occasionally, but as far as really putting my ideas out there in the open in my own blog to be trampled on, you know there's a bit of fear there I think that I have and so that has been difficult for me, to really put my ideas out there, which I know is good from the aspect of I could get feedback and learn from that, but there's still something in me that says what do I know? Who am I to contribute my thoughts to the world? I'm just this little person over here." (Participant 12)

Another member of this passive group failed to see the inherent value of learning through the network but instead seemed to be looking for a more formal course:

"I selected the MOOC because it was being led by a well-known, well established, very experienced names [sic] and that's actually the secret, I realize now that what I'm looking for when I'm looking to make a jump in development is actually more guidance rather than freeform learning. The whole point is I can do the freeform anywhere, anytime, 24 hours a day on the net." (Participant 23)

A third member expressed her frustration with not connecting: "I'm not really sure how to find a group of people online who really want to learn about what I most want to learn about" (Participant 13).

The invitation to participate in this study was sent out in Week 17 of the course. It is therefore surprising that these passive participants had persisted with the course to this point; perhaps they could be categorized as dissatisfied lurkers (in contrast to the satisfied lurkers identified above). Despite extensive efforts by the course organizers to accommodate learners of all types in the Change11 cMOOC, via videos and guidance in the orientation week of the course, it seems that the cMOOC format is not suitable for every learner. It is also important to acknowledge that it is likely that many course participants had dropped out by Week 17, and that it is this category of passive participants – whose needs are clearly not being met – who are most likely to have dropped out and are therefore underrepresented in this study.

Factors Affecting Engagement

Our second research question asked, "What principal factors mediate engagement?" From the accounts of engagement in the Change11 cMOOC that were collected, some key factors affecting engagement in this cMOOC can be identified:

Confidence. One key factor evident in the responses of both passive participants and lurkers relates to their confidence levels. Here, for example, a participant describes how her lack of confidence discouraged her from sharing her bookmarks socially:

"I actually did sign up with Diigo thinking right, this is it, I'm going to throw myself in. But once I was in the MOOC and running the idea, I actually felt almost as though I would be, instead of sharing with others, I was going to be simply showing my ignorance, even in terms of what I was selecting and why I was selecting it." (Participant 23)

Elsewhere, numerous participants described writing blog posts but not publishing them – in this way they gained the benefit of working through ideas, without having to expose those ideas to a potentially critical audience. Prior experience. An important factor was participants' prior experience. Twenty of 29 participants had previously participated in a cMOOC. As shown in Table 2, all but one of the active participants had previously participated in another cMOOC, while none of the four passive participants had previously participated in a cMOOC. Learning in a cMOOC is fundamentally different from learning in a formal course, and requires some adjustment.Table 2. Participant group by previous cMOOC experience

Participation in Previous cMOOCsMotivation. Finally, motivation was identified as an important determinant of engagement. Several of the active participants described a clear aim associated with their participation in the Change11 cMOOC. For example, this participant had clear ambitions to change his/her practice: "The ultimate aim of participating in this MOOC is to see how I can completely change the way that I teach and illustrate through example how others could do the same" (Participant 32). In contrast, passive participants had less well formed aims, as illustrated by this quote:

"Maybe that's part of my problem that I didn't maybe have a strong enough particular aim, like I'm interested in the idea of understanding better how they work, but I'm actually more interested in learning about other things that are nothing to do with online learning." (Participant 23)

Discussion and Conclusion

Active participants represent the key group in a cMOOC. A successful course becomes the content that these participants create and share, far more so than the live presentations and course readings. Moreover, the network of connections with other learners is something that can persist long after a cMOOC has ended. The more active or experienced members of the group provide a model for those who are less experienced, and are instrumental in creating the emergent spaces supporting connectedness and interactivity (Downes, 2009), which Kop et al. (2011) have argued are essential to successful connectivist learning environments. One key observation from this study is that all but one of the active participants had previously participated in at least one other cMOOC, while none of the four passive participants had participated in a cMOOC. It seems clear that learners must learn how to learn in a cMOOC.

In contrast to active participants, lurkers can potentially gain all the benefits of the course, but apparently contribute nothing in return. Discussing lurking in traditional online courses, Rovai (2000) describes lurkers as "learners who are bystanders to course discussions, lack commitment to the community, and receive benefits without giving anything back" (p. 291). This negative perception of lurkers was shared by some active participants who felt that being a lurker was incompatible with the concept of a cMOOC course, reliant as it is on participation and the activities of aggregation, relation, creation, and sharing identified by Kop (2011). However, cMOOCs must accommodate learners of all types to satisfy Downes' (2009) diversity and openness criteria. In practice, as long as there is a balance of these different types of learners, then lurkers can be accommodated, and the evidence from this study is that lurkers can learn effectively in connectivist environments: taking the knowledge they acquire to their own external networks.

In this study, the distinction between lurkers and passive participants reflects the observation that lurkers (most, but not all of whom self-identified as lurkers) were content with their participation in the cMOOC. In contrast, passive participants seemed frustrated with the course and their behavior indicates that they did not want the autonomy to choose where, when, how, and with whom to learn (Downes, 2009). Learner confidence was also a factor in this group, with some participants indicating that they felt they did not have anything useful to say. Others seemed to be lacking the critical literacies to learn effectively in a connectivist course.

The findings presented here are part of a larger study examining self-regulation of learning in cMOOCs. While the whole study contributes a substantial body of new empirical data about the learning experience afforded by cMOOCs, there are some limitations that must be acknowledged and considered. First, the sampling strategy used (recruitment via a call for participation included in the e-mail newsletter in Week 17 of the course) would not have reached participants who had already dropped out by this stage. Second, the authors noted that participants in this cMOOC were as interested in the cMOOC process, as in the Change11 content, and therefore it may be difficult to generalize the findings of this study to other courses. Careful design of future studies could avoid these limitations. Future studies should seek to compare the learning experience offered by different cMOOCs, target specific types of learners, and could attempt to follow learners across different cMOOCs to gain a better understanding of how critical literacies for learning in cMOOCs develop.

Understanding the nature of learners and their engagement is critical to the success of any online education provision, especially those where there is an expectation that the learner should self-motivate and self-direct their learning. Massive courses, by their very nature, bring in learners with a range of backgrounds, previous experience, and skill levels, and it is therefore incumbent on course organizers to design a learning experience that accommodates these diverse learner profiles. The three factors affecting engagement identified in this study (prior cMOOC experience, confidence, and motivation) provide an insight into how organizers of future cMOOCs might address this design challenge. Those participants who have not previously studied on connectivist courses can easily be identified. These participants may be given additional induction, or could be paired with a more experienced student who could act as a mentor. An approach like this would have helped Participant 13, who felt unable to find similar learners in the course. Those participants who lacked confidence could be paired with learners of similar experience to act as "buddies." Finally, learners could be encouraged to identify and articulate clear aims and goals for the course to increase motivation (Locke & Latham, 2002). Goals could also be used as a social object through which learners could find others with similar interests and aspirations (Milligan, Margaryan, & Littlejohn, 2012).

References

Anderson, T., & Dron, T. (2010). Three generations of distance education pedagogy. The International Review of Research in Open and Distance Learning,12(3), 80-97. Retrieved from http://www.irrodl.org/index.php/irrodl/article/view/890/1663

Barnard, L., Lan, W. Y., To, Y. M., Paton, V. O., & Lai, S.-L. (2009). Measuring self-regulation in online and blended learning environments. The Internet and Higher Education, 12(1), 1-6. doi:10.1016/j.iheduc.2008.10.005

Bouchard, P. (2009). Pedagogy without a teacher: What are the limits? International Journal of Self-Directed Learning, 6(2), 13-22. Retrieved from http://www.sdlglobal.com/IJSDL/IJSDL6.2-2009.pdf

Castells, M. (1996). The rise of the networked society. Oxford, UK: Blackwell.

Daniel, J. (2012). Making sense of MOOCs: Musings in a maze of myth, paradox and possibility. Journal of Interactive Media In Education, 2012(3). Retrieved from http://jime.open.ac.uk/article/2012-18/html

Downes, S. (2008). Places to go: Connectivism & connective knowledge. Innovate: Journal of Online Education, 5(1). Retrieved from http://www.innovateonline.info/pdf/vol5_issue1/Places_to_Go-__Connectivism_&_Connective_Knowledge.pdf

Downes, S. (2009, February 24). Connectivist dynamics in communities [Web log post]. Retrieved from http://halfanhour.blogspot.co.uk/2009/02/connectivist-dynamics-in-communities.html

Fini, A. (2009). The technological dimension of a massive open online course: The case of the CCK08 course tools. The International Review of Research in Open and Distance Learning, 10(5). Retrieved from http://www.irrodl.org/index.php/irrodl/article/view/643/1402

Garrison, D. R. (1997). Computer conferencing: The post-industrial age of distance education. Open Learning: The Journal of Open and Distance Learning, 12(2), 3-11. doi:10.1080/0268051970120202

Gijbels, D., Raemdonck, I., & Vervecken, D. (2010). Influencing work-related learning: The role of job characteristics and self-directed learning orientation in part-time vocational education. Vocations and Learning, 3(3), 239-255. doi:10.1007/s12186-010-9041-6

Guest, G., Bunce, A., & Johnson, L. (2006). How many interviews are enough? An experiment with data saturation and variability. Field Methods, 18(1), 59-82. doi:10.1177/1525822X05279903

Kanuka, H., & Anderson, T. (1999). Using constructivism in technology-mediated learning: Constructing order out of the chaos in the literature. Radical Pedagogy, 1(2). Retrieved from http://www.radicalpedagogy.org/Radical_Pedagogy/Using_Constructivism_in_TechnologyMediated_Learning__Constructing_Order_out_of_the_Chaos_in_the_Literature.html

Kop, R. (2011). The challenges to connectivist learning on open online networks: Learning experiences during a massive open online course. The International Review of Research in Open and Distance Learning,12(3), 19-38. Retrieved from http://www.irrodl.org/index.php/irrodl/article/view/882/1689

Kop, R., & Fournier, H. (2010). New dimensions to self-directed learning in an open networked learning environment. International Journal of Self-Directed Learning, 7(2), 1-20. Retrieved from http://www.sdlglobal.com/IJSDL/IJSDL7.2-2010.pdf

Kop, R., Fournier, H., & Mak, S. F. J. (2011). A pedagogy of abundance or a pedagogy for human beings: Participant support on massive open online courses. The International Review of Research in Open and Distance Learning,12(7), 74-93. Retrieved from http://www.irrodl.org/index.php/irrodl/article/view/1041/2025

Littlejohn, A., Milligan, C., & Margaryan, A. (2011). Collective learning in the workplace: Important knowledge sharing behaviours. International Journal of Advanced Corporate Learning, 4(4), 26-31. doi:10.3991%2Fijac.v4i4.1801

Locke, E. A., & Latham, G. P. (2002) Building a practically useful theory of goal setting and task motivation. American Psychologist, 57(9), 705-717. doi:10.1037/0003-066X.57.9.705

Mackness, J., Mak, S. F. J., & Williams, R. (2010). The ideals and reality of participating in a MOOC. In L. Dirckinck-Holmfeld, V. Hodgson, C. Jones, M. de Laat, D. McConnell, & T. Ryberg. (Eds.), Proceedings of the Seventh International Conference on Networked Learning (pp. 266-275). Lancaster, UK: University of Lancaster. Retrieved from http://www.lancaster.ac.uk/fss/organisations/netlc/past/nlc2010/abstracts/PDFs/Mackness.pdf

Maclellan, E., & Soden, R. (2006). Facilitating self-regulation in higher education through self-report. Learning Environments Research, 9(1), 95-110. doi:10.1007/s10984-005-9002-4

Milligan, C., Margaryan, A., & Littlejohn, A. (2012). Supporting goal formation, sharing and learning of knowledge workers. In A. Ravenscroft, S. Lindstaedt, C. Delgado Kloos, & D. Hernández-Leo (Eds.), 21st century learning for 21st century skills: Proceedings of the Seventh European Conference on Technology Enhanced Learning (EC-TEL 2012). Vol.  7563 Lecture Notes in Computer Science (pp. 519-524). Heidelberg, Germany: Springer. doi:10.1007/978-3-642-33263-0_53

Pintrich, P. R., Smith, D. A. F., García, T., & McKeachie, W. J. (1991). A manual for the use of the Motivated Strategies for Learning Questionnaire (MSLQ). Ann Arbor, MI: University of Michigan, National Center for Research to Improve Postsecondary Teaching and Learning. Available from ERIC database. (ED338122)

Rovai, A. P. (2000). Building and sustaining community in asynchronous learning networks. The Internet and Higher Education, 3(4), 285-297. doi:10.1016/S1096-7516(01)00037-9

Schraw, G., & Dennison, R. S. (1994). Assessing metacognitive awareness. Contemporary Educational Psychology, 19(4), 460-475. doi:10.1006/ceps.1994.1033

Sfard, A. (1998). On two metaphors for learning, and the dangers of choosing just one. Educational Researcher, 27(2), 4-13. doi:10.3102/0013189X027002004

Siemens, G. (2005). Connectivism: A learning theory for the digital age. International Journal of Instructional Technology and Distance Learning,2(1), 3-10. Retrieved from http://www.itdl.org/Journal/Jan_05/article01.htm

Siemens, G. (2012). MOOCs are really a platform [Web log post]. Retrieved from http://www.elearnspace.org/blog/2012/07/25/moocs-are-really-a-platform

Toering, T., Elferink-Gemser, M. T., Jonker, L., van Heuvelen, M. J. G., & Visscher, C. (2012). Measuring self-regulation in a learning context: Reliability and validity of the Self-Regulation of Learning Self-Report Scale (SRL-SRS). International Journal of Sport and Exercise Psychology, 10(1), 24-38. doi:10.1080/1612197X.2012.645132

Wilson, S., Liber, O., Beauvoir, P., Milligan, C., Johnson, M., & Sharples, P. (2007). Personal learning environments: Challenging the dominant design of educational systems. Journal of e-Learning and Knowledge Society, 3(2), 27-38. Retrieved from http://www.je-lks.org/ojs/index.php/Je-LKS_EN/
article/download/247/229

Acknowledgments

The authors would like to thank all those participants who gave their time for this study. They would also like to thank Lou McGill for conducting the interviews and Susan Houston for transcribing the audio of those interviews.

View the original article here

Learner Participation and Engagement in Open Online Courses: Insights from the Peer 2 Peer University

         Learner Participation and Engagement in Open Online Courses:
Insights from the Peer 2 Peer University

June Ahn
Assistant Professor
College of Information Studies and College of Education
University of Maryland, College Park
College Park, MD 20740 USA
juneahn@umd.edu

Brian S. Butler
Associate Professor
College of Information Studies and Robert H. Smith School of Business
University of Maryland, College Park
College Park, MD 20740 USA
bsbutler@umd.edu

Alisha Alam
Graduate Research Assistant
College of Information Studies
University of Maryland, College Park
College Park, MD 20740 USA
alisha@umd.edu

Sarah A. Webster
Graduate Research Assistant
College of Information Studies
University of Maryland, College Park
College Park, MD 20740 USA
websters@umd.edu

Abstract

Recent developments in massive open online courses (MOOCs) have brought renewed attention to online learning, but most of the attention has been centered on courses offered through platforms such as Coursera and edX. This paper focuses on the importance of examining alternative, large-scale learning initiatives that promote more participatory modes of education production and delivery. It presents a comprehensive description of the Peer 2 Peer University (P2PU), a social computing platform that promotes peer-created, peer-led, online learning environments. Using log data from the P2PU platform, the ecosystem of this learner-generated education platform is described. The descriptive analysis highlights P2PU's growth in terms of the participatory learning environments that have been created – such as online study groups, courses, and challenges – and also describes the participation patterns of P2PU members. This paper provides one of the first empirical descriptions of an emerging open learning platform and illuminates how log data from the platform, particularly in relation to open courses, open badges, and learning tasks embedded in courses, can be used to track development of courses and engagement in learning activities across P2PU. The analyses reported here are aimed at helping researchers understand the P2PU ecosystem and identify potential areas for future study as P2PU works to open its data for public analysis.

Keywords: massive open online course (MOOC), open education, peer-to-peer learning, participatory learning, crowdsourced education

Introduction

While recent platforms for massive open online courses (MOOCs) such as Coursera and edX have garnered substantial public attention, MOOCs were originally conceived of as participatory environments where groups of learners (from small to massive) collaborate to aggregate self-created learning experiences (McAuley, Stewart, Siemens, & Cormier, 2010). In this model of peer-based, open learning, a course might organize topics and guide the schedule. However, individual learners coordinate the learning activities by contributing blog posts, links to resources, and other media. This model brings to the foreground a bottom-up, crowd-generated approach to delivering online education versus the top-down approach implemented in popular platforms. Yet in spite of their differences, learner participation and engagement is a concern in both types of MOOCs (Waters, 2012).

At a basic level, any learning platform requires participant engagement to achieve its goals. Individuals must participate in the provided activities in order to have the learning experiences that are the focus of the platform. Beyond this basic form of participation, crowd-generated learning platforms require learners to engage in many other activities such as creating courses and learning materials, joining online courses, contributing comments and discussion, and persisting in the group's learning activities. Creating a participatory MOOC environment requires both more coordinated work (Butler & Ahn, 2013) and more types of participation than top-down MOOCs that rely primarily on individual learner engagement (Ahn, Weng, & Butler, 2013; Kafai & Peppler, 2011). However, in spite of its importance, there has been little empirical work done on conceptualizing and measuring participation in large scale, open learning platforms.

In the following study, the patterns of participation and engagement present in a prominent participatory, open online learning community, the Peer 2 Peer University (P2PU), are examined. An outline is first provided of how P2PU is an example of a peer-generated open online course platform that speaks to the original notion of participatory MOOCs. Then, statistics derived from a comprehensive dataset about P2PU courses and participant activity are presented, with the aim of exploring the question of how learners have participated and engaged with open online courses in P2PU.

The data in this study describe the course ecology that exists on the P2PU platform using metrics such as the number of member-created courses, types of courses, patterns of course enrollment, and active contributions to these peer-to-peer courses over time. A descriptive analysis of the diverse learning environments that have arisen in this participatory, open education platform is also offered.

This paper makes several contributions to research on open online learning communities. Recent developments in top-down MOOCs (Coursera, edX, etc.) have brought renewed energy and attention to online learning. It highlights the importance of examining alternative, and equally exciting, open learning initiatives such as P2PU. This paper provides the first empirical descriptions of an emerging open learning platform, illuminating how log data from P2PU, particularly in relation to open courses, open badges, and learning tasks embedded in courses, can be used to track the development of courses and engagement in learning activities across P2PU. Different flavors of online education are developing at a rapid pace, bringing to the fore numerous research questions and concerns. The descriptions presented in this paper are aimed to help researchers understand the P2PU ecosystem and identify potential areas for future study as P2PU works to open its data for public analysis.

Related Work

Understanding the different definitions of the term MOOC is vital to develop a nuanced understanding of how educational experiences are developed and delivered in various online platforms. Recent, high profile MOOC examples such as Coursera, edX, and Udacity represent what some term "xMOOCs," with their institutional, top-down, content-delivery driven models (Siemens, 2012). Courses in these platforms are typically designed as a weekly syllabus of video lectures followed by quizzes or other assignments that evaluate whether students understood the content. Discussion boards are also often provided for students to ask questions and seek clarification on information related to the course. These courses mirror the structure of what was seen in most online education platforms. The major innovations have been the ability of these platforms to curate and freely deliver content from elite universities and garner "massive" enrollments of thousands and tens of thousands of learners.

In contrast to this traditional model of teaching and learning is what educators have referred to as connectivist MOOCs or "cMOOCs" (McAuley et al., 2010; Siemens, 2012). This original form of MOOC was viewed as an experiment that leveraged the distributed, networked information available on the Internet with the learners themselves generating most of the learning activities. Learners might write their own reflections about what they are learning on blogs and other social media platforms, share these back with their peers, create social networks with others in the MOOC, share information, and aggregate their ideas to create an emergent learning experience.

This form of collective learning differs from the xMOOC model in several significant ways (Kafai & Peppler, 2011). First, the modes of production differ dramatically. In xMOOCs, elite institutions such as Stanford University, Harvard University, Massachusetts Institute of Technology, and partner universities develop courses. In cMOOCs, the underlying philosophy is that any individual should be empowered to design a learning experience. Second, course implementation and the role of learners are conceptualized very differently. In an xMOOC, learner responsibilities focus on consuming the course content and completing evaluations to assess understanding of that content. In a cMOOC, learners may utilize the course-provided content, but as one resource among many that they may access through their own online searching and research. Learners take on the dual role of learners and teachers, as they help their peers through the learning process they are also undertaking. In a cMOOC, the content of the course is user-generated and emergent, arising from the persistent contributions of the learners themselves. Leadership is distributed and the learning experience is severely limited if there are only a few students generating ideas and information.

Connectivist or collectivist MOOC environments build on broader trends in computing and technology-mediated social interaction. These versions of MOOCs leverage the capabilities of social computing, where networked media and technology enable individuals to easily share information, communicate with others, and drive online content-creation (Parameswaran & Whinston, 2007). Thus, instead of treating learning as information transfer from instructor to student, learning is conceptualized as arising from social interaction and information sharing via networked tools. A collectivist philosophy of online education also represents a particular instantiation of crowdsourcing. The term crowdsourcing describes the use of technology to outsource a task typically done by a particular agent or person by distributing the task to a large group of people to do collectively (Howe, 2006). In the MOOC context, one could observe the process of course creation as a crowdsourcing example. In xMOOC platforms, course creation is crowdsourced, in a sense, to individual universities and their instructors, but aggregated in a central platform. Yet at a micro-level, courses are still designed and created by a single source, the instructor. In a cMOOC context, the course content and design are further crowdsourced to the learners themselves. Rather than one source of information and content, learners form online social networks in which they share and aggregate content across multiple channels. Through the use of social computing platforms, education could potentially be a crowdsourced endeavor, where learners create, manage, and collectively implement large-scale, online courses.

The Peer 2 Peer University

P2PU is non-profit organization and online platform that began in 2009. The online community promotes experimentation in open learning and peer-led education. The platform allows any member to design and create an educational course, which can then be taken by any other member in the online community. In P2PU, learning environments come in a variety of formats (with courses being only one instantiation). The general term used in P2PU is learning "project," and this term will be used in the present paper to speak generally to different P2PU learning situations. More specific terms refer to particular learning environments. In P2PU, members can create study groups to bring together learners around topics of interest for a set period of time. Members can also create more formal courses that are structurally similar to study groups. The most recent iteration of P2PU adds challenges, which are asynchronous courses that remain persistent and act as guides for learners through an educational experience (rather than being time-bound). Learners in P2PU can join, complete, and leave challenges at any moment. P2PU has also introduced new forms of collective assessment through open badges. Learners can earn badges associated with different learning tasks and courses, which tie into the Mozilla Open Badges framework (The Mozilla Foundation & P2PU, 2012). Study groups, courses, challenges, and badges are all important constructs that P2PU provides to organize peer-generated learning experiences, and each will be considered in the analysis presented below.

In many ways, P2PU is a social computing platform that crowdsources the creation and implementation of informal online education. Instead of formal teachers or established institutions acting as the sole developers of online courses, any stakeholder can create a course in P2PU. In addition, the mission and ethos of the P2PU community encourages a social process of learning. Courses are not designed merely as content delivery mechanisms, but as spaces that ask members to engage in active learning projects, share their progress and resources, start and participate in discussions, and collaborate to gain knowledge of a topic area. Similarly, P2PU's experiments with open badges are a way to observe how social assessments of learning can be recognized and credited in open, informal learning environments. As a crowdsourced education platform, the success of the community is critically dependent on each member's active and persistent participation.

In this study, the authors sought to better understand the notions of learner participation and engagement in open online courses, and explore how researchers could observe these activities using log and trace data from P2PU. The goal was to take the raw data provided by P2PU, and use descriptive analytics to understand the broader ecosystem of the P2PU community. The main research question guiding this study was:

How have learners participated and engaged with open online learning in P2PU?

In a participatory, open education setting like P2PU, the learning environments themselves are not givens. There is a substantial amount of member effort needed to create the basic learning contexts (e.g., courses and challenges) and assessments (e.g., badges) required for the platform to function. As Butler and Ahn (2013) observe, a great amount of cooperative work and coordination is needed to create materials, mobilize peers, and collaborate in productive ways, before learning can even occur. Thus, the descriptive analysis of the P2PU log data began with an exploration of exploring questions such as:

How many courses and challenges comprise the P2PU community?What patterns of participation can be observed in these learning contexts?What are the characteristics of learning environments within P2PU?How can learners' participation and engagement in these contexts be measured?These descriptive analyses of the P2PU community can then inform future researchers seeking to explore complex relationships between learning context characteristics and learner participation, and engagement.

Method

The study reported here was part of an ongoing partnership with P2PU, in which the main goals are to: (1) create policies and processes to publicly share datasets of the P2PU platform; and (2) conduct analyses of this widely used open education platform to shed light on the challenges of supporting large-scale, participatory, online learning. The project includes substantial data work to transform raw P2PU log data into usable variables and datasets for researchers in education and other related fields. In addition, the research team is exploring various learning analytics techniques for illuminating the learning processes that occur in this online community (Bienkowski, Feng, & Means, 2012).

In October 2012, P2PU provided the research team with a raw dataset containing a complete record of all elements of the platform (courses, challenges, users, learning activities, etc.). This dataset covered the entire history of P2PU from its beginning in late 2009 to October 2012. The data was a copy of the back-end mySQL database that is the operational foundation for P2PU.org and consisted of 82 tables. In an initial phase of data exploration, the research team developed a high-level mapping of the tables that corresponded to the primary components of the P2PU platform and their relationships to one another (Figure 1). This study utilized log data from those tables that described P2PU Schools, Projects (e.g., the different types of learning contexts), Badges, Users, and user Content contributions.

Figure 1. High-level description of the P2PU relational database structure

Although the database theoretically provided a rich, fine-grained record of P2PU schools, projects, users, their activity, and badges, the raw log data was structured in terms of operational transactions and entity status records which did not readily correspond to the constructs most likely to be of interest to researchers and analysts. To overcome this mismatch the research team first conducted a phase of data exploration and descriptive analytics to construct a holistic understanding of various levels of the P2PU community. Then on this basis, a project-level dataset was created that characterized the nature, history, and activity associated with every project (e.g., study group, course, challenge) ever created on the P2PU platform. This analysis dataset included a number of variables for each project, which are described in turn below.

Project name: Name of the P2PU project. Language: What language the project uses (e.g., English, Spanish). Project category: What type of project is it (study group, course, or challenge)? Under development: A binary flag that designates whether a given project is under development or "live" and available to the public. Deleted: A binary flag designating whether a project has been deleted. Archived: A binary flag designating whether a project is no longer publicly available and has been archived. Project pages: A count of how many website pages have been created for a given project. Organizer count: P2PU project may have one or more creators or leaders (known as organizers). This variable indicates how many unique organizers are associated with a project. Participant count: P2PU Members can join study groups and courses as active "participants." This variable is a count of how many members have joined and self-designated as participants. Follower count: Members can also join study groups and courses as "followers," indicating that they will not actively participate but are interested in receiving notices about activity within the course. This variable is a count of how many members have joined and self-designated as followers. Total number of tasks: Each challenge consists of a series of tasks to be completed by learners. As they complete them, members record their progress by checking off each task. This variable indicates how many tasks are associated with each challenge. Adopter count: In challenges, members can join as active participants by working on tasks in order to complete the challenge. This variable indicates how many members begun working on the tasks associated with a challenge. Completers: In challenges, members can check off tasks as they work to achieve the challenge. When all the tasks are marked as completed, the challenge is said to have been completed. This variable is a count of how many members have completed a challenge. Average tasks completed: While some individuals complete all the tasks associated with a challenge, many do not. This variable captures the average number of tasks completed by the adopters of each challenge, providing a metric of the overall progress of the members participating in it. Total number of badges: Some challenges and courses are associated with badges that members can earn if they complete certain tasks. This variable is a count of how many badges were associated with a given project. Average badges earned: As with tasks, individuals can vary with respect to how far they have progressed with respect to the badges associated with a project. This variable captures the average number of badges completed (from those associated with the project) among the participating members.These variables were constructed based on an examination of the P2PU interface, consultation with P2PU developers, and detailed analysis of the P2PU platform code, which is available through an open source license. The constructed measures were also validated with SQL queries to multiple combinations of data tables of P2PU's backend database.

Findings

The P2PU Ecosystem

To explore how learners have participated and engaged with open online learning in P2PU several platform-level metrics that provided a glimpse into P2PU's history, its ecosystem of learning projects, and members in the community were examined. The primary component of the P2PU learning ecosystem is the project. Projects are bounded collections of dynamic and static content, organizers, participants, and infrastructure. There were 2,034 project records in the P2PU database that are broken down into three different learning environment types: 1,135 study groups, 506 courses, and 393 challenges.

Study groups bring together groups of informal learners together around educational tasks and discussions. Courses are similar in structure to study groups, but represent more formal learning projects that are open to anyone in P2PU. Courses organize around tasks that learners complete (and are typically structured as weekly tasks), in addition to threaded discussion pages where learners can participate in dialogue and share thoughts. In the recent iteration of P2PU, another learning environment called challenges was introduced. Challenges utilize a similar structure as courses but are asynchronous and persistent (whereas courses ran for a set period of time and learners dispersed). A challenge is a series of tasks and discussion spaces, where a learner can enter and exit at any point. Badges can also be associated with these challenges, which learners earn by completing various tasks. In approximately two years, P2PU has supported the creation of just over 2,000 distinct learning environments that combine a set of common elements to provide rich learning experiences for participants.

While there are numerous projects in P2PU, they vary significantly with respect to their level of development. The majority of the over 2,000 projects in P2PU were tagged as "under development," which means that members experimented with creating these learning environments, but never implemented them live in the community. Most projects appear to have started development but were never finished and released live, and others were deleted from the site over time. Of the total, there were 368 projects (18.09%) that went live and were not deleted from the P2PU platform. Of these projects 159 were challenges, 132 were study groups, and 77 were courses. About 85% of these projects were in English and 12% in Spanish (with less than 3% in other languages).

At the same time, it was also observed that in P2PU's nascent history, there has been a general trend of 10 to 25 new learning projects being developed and publicly released each month, with a sharp rise in July 2012 due to a special initiative that month (see Figure 2 below). Together, these statistics suggest that as a platform P2PU is able to successfully encourage individuals to experiment with creating a diverse array of participatory learning environments, but that a relatively small percentage of those projects are ever implemented as operational. Future research should consider the issues related to the development of learning projects in P2PU, strategies for fostering more complete learning project development over time, and analysis of whether this ratio of experimentation to "published" projects is optimal. In addition, there are rich opportunities to understand the barriers that individuals may face in the creation of projects within such a participatory platform. Factors such as a lack of understanding of the P2PU platform, prior knowledge in topic areas, motivation, or lack of community involvement may influence the systematic production and availability of learning projects over time.

Figure 2. Projects created by month ("live" courses only)

Along with this steady development of peer-created learning environments, there were 41,281 registered members in P2PU at the time of writing of this paper. Such membership represents a truly massive group of potential learners, and positions P2PU as an example of a large-scale participatory learning environment. P2PU does not describe itself as a cMOOC environment, but rather a platform to promote open, informal learning. However, taking a broad view of the entire platform, it is an online community of over 40,000 potential learners who can share and create their own participatory learning environments and experiences. This large pool of potential learners is notable as an experiment in participatory, peer-led learning; nonetheless, participation and engagement issues remain, just as in any other online community. Data in the P2PU database provides time-stamped fields for when a member created their P2PU account and the last date they were active in the community (when they logged on). Using these fields, it is possible to create various overall participation metrics, such as how many P2PU members ever returned to the site after first creating their account. It was found that 6,483 members returned to P2PU at least once after account creation (approximately 16% of all registered members). This implies that that the majority of members (approximately 85%) created accounts and never again engaged with the community. While this pattern of overall engagement (or relatively lack thereof) is a potential problem for P2PU, it is consistent with participation patterns observed in other online environments (Cummings, Butler, & Kraut, 2002). Nevertheless, having over 6,000 active members in this community still represents a substantial pool of learners and contributors in this participatory learning community. Furthermore, as P2PU has continued to establish itself as an online community, one can also observe that they have consistently attracted 300-400 new members per month (who return to the site) over the 2012 year (see Figure 3). Overall, these descriptive measures of P2PU's growth and history show an online community that is in an early stage of development, but steadily growing both in terms of individual users and different types of participatory learning environments.

Figure 3. Member accounts created by month (for 6,483 members who have returned at least once to P2PU)

To explore the relationship between these two interdependent elements of the P2PU ecology, metrics that describe how members participated and engaged within the context of study groups, courses, and challenges were examined. Members in P2PU can relate to projects in various ways. They can organize and create projects, or they can join in on projects as participants or followers. Of the 6,483 members of P2PU who logged in at least once more after creating an account, 4,730 (73%) have signed up to be an organizer, participant, or follower of at least one learning project. This suggests that of P2PU members who do return to the site, a substantial number attempt to become active members in projects by organizing a project or signing up to be a participant.

There was also significant variation in the number of members participating in different learning environments both within and across the different types (see Table 1). For example, study groups and courses in P2PU typically had 1-2 organizers who created the learning group and initiated activities. The majority of these learning groups were small. The median number of participants in study groups was three members and the median in courses was five members. However, some study groups and courses grew into quite large learning environments. Study groups reached up to 370 participants and 1,167 followers (individuals who followed along but did not actively participate). The largest courses grew to 173 members and 319 followers.

Table 1. Number of members who participated in P2PU study groups, courses, and challenges (live projects only)

Membership in P2PU challenges is organized a little differently. Learners choose to adopt the challenge and the platform also keeps track of who has completed a given learning challenge (completers). Again, a majority of challenges are small, with the median number of adopters being two members – although the distribution is skewed with some challenges successfully having up to 158 (current) adopters and 694 completers. These descriptive statistics highlight the wide diversity of participatory learning environments that arise in P2PU. Many study groups, courses, and challenges attract only a few like-minded individuals. However, some projects attract hundreds of learners to create larger-scale learning environments. Again, while the skewed distribution of participation potentially presents a challenge for P2PU and other open learning environments, it is consistent with the distribution of group and forum sizes that has been observed in other online contexts (Butler, 2001; Butler & Wang, 2012; Jones, Ravid, & Rafaeli, 2004). Future research that explores the relationships between support for development of focused learning environments, characteristics of the created learning environments, and the resulting patterns of participation would make substantial contributions to understanding how open learning communities such as P2PU can function more effectively.

Tasks and Badges: Opportune Measures of Engagement

As the P2PU platform has developed, support for different types of learning environments have been added. Initially, study groups (time-bounded, unstructured group interaction spaces) were supported, then courses (time-bounded, guided exploration of topics, with associated group interaction). Most recently challenges were introduced as ways to organize learning. Challenges are organized as an informal syllabus, consisting of a checklist of tasks. P2PU members join challenges and when they complete various learning tasks, they check off these tasks as completed. One unique affordance of this feature is that P2PU logs a record of challenges, tasks, adopters, and when these adopters checked off tasks as complete. Using these logs, measures of the persistence and engagement of participants in P2PU challenges as of October 2012 could be calculated.

The detailed logging of tasks provides some nuanced ways to conceptualize and observe persistence and engagement. For example, challenges in P2PU had an average of 4.67 learning tasks that members were asked to complete (Table 2). On average, members reported completing roughly half of the available tasks (2.24 average number of tasks completed per adopter). Interesting future research streams emerge from this descriptive data. For example, using the P2PU data it is possible to examine longitudinal trends of when P2PU members complete tasks and how other variables influence this task completion. In addition, future research might examine how characteristics of challenges, tasks, and P2PU members and their social networks relate to the completion of tasks over time.

Table 2. Tasks and badges associated with challenges in P2PU

Average number of tasks completed per adopterAverage number of badges earned per member

Note. N = 159 active challenges in P2PU.

In addition to tasks, P2PU had recently begun piloting an open badge system based on the Mozilla Open Badge Framework (The Mozilla Foundation & P2PU, 2012). The idea behind open badges is to provide publicly visible credentials for informal learning, and there is a rich area of future research to examine the varied social, cultural, and educational functions of online badges and informal credentialing (Abramovich, Schunn, & Higashi, 2013; Antin & Churchill, 2011). As with study groups, courses, and challenges, members of P2PU can create badges that other members can earn by completing different learning tasks and challenges. Badges are also logged in the P2PU system and provide a way to measure and understand learner achievements and their engagement with learning over time.

As of October 2012, the implementation of badges was in an early stage. As a result, many challenges did not have associated badges (i.e., challenges had an average of less than one badge associated with them). Some challenges had up to eight associated badges with some activity in members earning badges (Table 2). In future research, some interesting avenues could include examining learners' trajectories of badge earning and the different social dynamics at play in P2PU in relation to badge earning (e.g., to better understand who creates and earns badges, as well as what the predictors badge earning are).

Conclusion

One lesson from the recent rise of MOOCs (Pappano, 2012) is that new forms of online education will most likely endure, evolve, and become a vital part of how education is delivered in the future. Much of the initial excitement and attention to MOOCs have risen from a particular version of online learning, the xMOOC. This paper has highlighted how alternative, participatory forms of education production and delivery can develop within and be supported by social platforms such as P2PU. In such participatory, collective cMOOC contexts, participation and engagement take on varied forms. Courses must be developed, assessments created, fellow learners recruited, and learning environments sustained over time through engagement and participation. These different functions bring about new challenges to sustaining robust, large-scale, participatory learning environments. The challenges illuminate potentially high-impact research areas that could make a substantial contribution to improving and evolving large-scale, online learning that are alternatives to the high-profile examples that are popular today (Coursera, etc.).

Through the descriptive analysis of P2PU presented in this paper, some of these challenges have been highlighted and potential research directions for future work illuminated. P2PU is clearly an early-stage, but growing, online community. From the log data, it is evident that steady streams of members are joining the community over time, and that these members are actively experimenting with creating their own learning environments online. However, a substantial number of learning projects on P2PU are never released live and implemented. Interesting future questions remain about the experiences of learners who take on the role of course creators, including the following:

What are their motivations?What challenges do they face in creating participatory learning environments?What skills are needed to create effective courses, such as instructional design, and more importantly, how can the learning of these skills be "scaled out" to everyday individuals to foster more effective crowdsourced course development?What factors are related to courses actually being implemented in P2PU?Research in this stream would shed light on issues such as new forms of organization, instructional design, and production of education in a participatory, open, social computing context.

The analysis also highlights the wide variability present in the online learning environments that are present in P2PU. In a participatory setting such as P2PU, the majority of learning projects serve small groups of informal learners who come together around specific topics. These learning arrangements represent the long-tail of the ecosystem of learning contexts in P2PU with many courses that serve a few individuals around niche interests (Brown & Adler, 2008). A small number of P2PU projects successfully garnered the participation of hundreds and thousands of interested learners, and could be viewed as a form of cMOOC. Individual P2PU projects that garner thousands of students introduce new issues of educational practice at scale. However, the aggregation of hundreds and thousands of smaller learning arrangements in a platform such as P2PU also combines to create an alternative, and uniquely large-scale, ecology of education opportunities. Future questions remain concerning how to foster robust learning communities from this ecological perspective (Butler & Ahn, 2013). How many courses and members are needed to foster participation and learning activities? What factors within courses or learning projects are related to fostering engagement over time (e.g., Ahn, Weng, & Butler, 2013)? How can platforms such as P2PU foster the development of participatory learning behaviors such as social network formation, knowledge sharing, and active learning opportunities?

Finally, the P2PU platform also affords opportunities to examine concepts such as participation and engagement through new artifacts such as online learning tasks and open badges. It has been demonstrated here how log data of learning tasks in P2PU can show the variability of learning activity conducted by members. Future work is possible to examine longitudinal patterns of the learning tasks P2PU members undertake, which may provide indicators of sustained participation and engagement. In addition, research that can examine the relationship between factors such as course design, social interaction, and networks with the completion of learning tasks over time can make a substantial contribution to fundamental questions around supporting online learners to persist and not drop out of courses over time. P2PU's early piloting of open badges, and their logging of badge creation and earning behaviors, can also serve as an exciting context to explore the social, cultural, and educational functions of badges in online learning. These research areas promise to rise in importance as badges, and other artifacts of informal educational credentialing, become a major element of online learning in the future.

Overall, P2PU provides a rich environment for addressing many fundamental questions about the development, operation, and use of large scale, open, online learning platforms. While this work is still in its early stages, it provides an initial look at the way that focused learning environments, such as P2PU study groups, courses, and challenges, are developed (or not), are adopted (or not), and are engaged by learners (or not). Realizing the potential of MOOCs and other forms of large scale, technology-enabled learning environments will depend not only on understanding how to provide appropriate experiences for individual learners, but also on our ability to design platforms that provide the affordances necessary to support development of diverse populations of focused learning environments.

References

Abramovich, S., Schunn, C., & Higashi, R. M. (2013). Are badges useful in education? It depends upon the type of badge and expertise of learner. Educational Technology Research & Development, 61(2), 217-232. doi:10.1007/s11423-013-9289-2

Ahn, J., Weng, C., & Butler, B. S. (2013). The dynamics of open, peer-to-peer learning: What factors influence participation in the P2P University? In R. H. Sprague, Jr. (Ed.), Proceedings of the 46th Annual Hawaii International Conference on System Sciences (pp. 3098-3107). Los Alamitos, CA: IEEE Computer Society. doi:10.1109/HICSS.2013.515

Antin, J., & Churchill, E. F. (2011, May). Badges in social media: A social psychological perspective. Paper presented at the ACM SIGCHI Conference on Human Factors in Computing Systems, Vancouver, Canada. Retrieved from http://labs.yahoo.com/files/Antin%20&%20Churchill%20-%20Badges%20in%20Social%20Media.pdf

Bienkowski, M., Feng, M., & Means, B. (2012). Enhancing teaching and learning through educational data mining and learning analytics: An issue brief. Washington, DC: U.S. Department of Education, Office of Educational Technology. Retrieved from http://www.ed.gov/edblogs/technology/files/2012/03/edm-la-brief.pdf

Brown, J. S., & Adler, R. P. (2008). Minds on fire: Open education, the long tail, and Learning 2.0. EDUCAUSE Review, 43(1), 16-32. Retrieved from http://www-cdn.educause.edu/ir/library/pdf/ERM0811.pdf

Butler, B. S. (2001). Membership size, communication activity, and sustainability: A resource-based model of online social structures. Information Systems Research, 12(4), 346-362. doi:10.1287/isre.12.4.346.9703

Butler, B. S., & Ahn, J. (2013, February). Ecological perspectives on creating and sustaining open learning environments. Paper presented as part of the Workshop on CSCW and Education at the 16th ACM Conference on Computer Supported Cooperative Work, San Antonio, TX. Retrieved from http://www.ahnjune.com/wp-content/uploads/2012/12/CSCW-Position-Paper-_final.pdf

Butler, B. S., & Wang, X. (2012). The cross-purposes of cross-posting: Boundary reshaping behavior in online discussion communities. Information Systems Research, 23(3), 993-1010. doi:10.1287/isre.1110.0378

Cummings, J. N., Butler, B., & Kraut, R. (2002). The quality of online social relationships. Communications of the ACM, 45(7), 103-108. doi:10.1145/514236.514242

Howe, J. (June 2, 2006). Crowdsourcing: A definition [Web log post]. Retrieved from http://crowdsourcing.typepad.com/cs/2006/06/crowdsourcing_a.html

Jones, Q., Ravid, G., & Rafaeli, S. (2004). Information overload and the message dynamics of online interaction spaces: A theoretical model and empirical exploration. Information Systems Research, 15(2), 194-210. doi:10.1287/isre.1040.0023

Kafai, Y. B., & Peppler, K. A. (2011). Beyond small groups: New opportunities for research in computer-supported collective learning. In H. Spada, G. Stahl, N. Miyake, & N. Law (Eds.), Connecting computer-supported collaborative learning to policy and practice. Proceedings of the Ninth International Conference on Computer-Supported Collaborative Learning (CSCL 2011) (Vol. 3, pp. 17-24). Atlanta, GA: International Society of the Learning Sciences. Retrieved from http://www.gerrystahl.net/proceedings/cscl2011/cscl2011proceedingsIII.pdf

McAuley, A., Stewart, B., Siemens, G., & Cormier, D. (2010). The MOOC model for digital practice. Charlottetown, Canada: University of Prince Edward Island. Retrieved from http://www.elearnspace.org/Articles/MOOC_Final.pdf

The Mozilla Foundation, & Peer to Peer University (with The MacArthur Foundation). (2012). Open badges for lifelong learning. Retrieved from https://wiki.mozilla.org/images/b/b1/OpenBadges-Working-Paper_092011.pdf

Pappano, L. (2012, November 2). The year of the MOOC. The New York Times, ED26. Retrieved from http://www.nytimes.com/2012/11/04/education/edlife/massive-open-online-courses-are-multiplying-at-a-rapid-pace.html

Parameswaran, M., & Whinston, A. B. (2007). Social computing: An overview. Communications of the Association for Information Systems, 19, 762-780. Available from the AIS Electronic Library. (http://aisel.aisnet.org/cais/vol20/iss1/1)

Siemens, G. (2012, March 5). MOOCs for the win! [Web log post]. Retrieved from http://www.elearnspace.org/blog/2012/03/05/moocs-for-the-win/

Waters, A. (2012, July 23). Dropping out of MOOCs: Is it really ok? [Web log post]. Retrieved from http://www.insidehighered.com/blogs/hack-higher-education/dropping-out-moocs-it-really-okay

View the original article here