MIT and Harvard release working papers on open online courses

MIT and Harvard University today announced the release of a series of working papers based on 17 online courses offered on the edX platform. Run in 2012 and 2013, the courses analyzed drew upon diverse topics — from ancient Greek poetry to electromagnetism — and an array of disciplines, from public health to engineering to law.

The working paper series features detailed reports about individual courses; these reports reveal differences and commonalities among massive open online courses (MOOCs). In the coming weeks, data sets and interactive visualization tools will also be made available.

Led by Isaac Chuang, a professor of electrical engineering and computer science at MIT, and Andrew Ho, an associate professor in Harvard’s Graduate School of Education, the collaborative research effort was in service of a mutual goal — “to research how students learn and how technologies can facilitate effective teaching both on-campus and online” — part of a mission statement established when MIT and Harvard joined to form edX, the not-for-profit online learning platform, in May 2012.

The papers analyze an average of 20 gigabytes of data per course and draw on interviews with faculty and course teams as well as student metrics.

Key takeaways

Takeaway 1: Course completion rates, often seen as a bellwether for MOOCs, can be misleading and may at times be counterproductive indicators of the impact and potential of open online courses.

The researchers found evidence of large numbers of registrants who may not have completed a course, but who still accessed substantial amounts of course content. Across the 17 MITx and HarvardX courses covered in the reports, 43,196 registrants earned certificates of completion. Additionally, another 35,937 registrants explored half or more of the units in a course without achieving certification.

“We found students in the courses who engaged with every single piece of the courseware, students who only read text or viewed videos, students who only took assessments or completed problem sets, and students representing nearly every possible combination of these behaviors,” Chuang explains. “Experimentation is part of the learning process.”

An additional 469,702 registrants viewed less than half of the units in a course and another 292,852 registrants never engaged with the online content. In total, the 17 online courses drew 841,687 registrations from 597,692 unique users.

The reports note that many faculty encouraged learners to participate in the course experiences in whatever ways they found useful, whether or not that involved trying to earn a certificate. For such faculty, the participation of engaged auditors was just as important as the participation of those who completed the assessments of the course.

“A fixation on completion rates limits our imagination of what might be possible with MOOCs. A better criterion for success might be for students to complete more of the course than they thought they would, or to learn more than they might have expected when they first clicked on a video or course forum,” Ho says.

Takeaway 2: Most MOOC attrition happened after students first registered for a course. On average, 50 percent of people left within a week or two of enrolling. After that window, attrition rates decreased substantially. The average probability of a student ceasing to engage in the second week of the course declined to 16 percent.

While the persistence rates in MOOCs look very different from those of conventional courses in higher education, they look very similar to how people interact with other Web-based media, such as video or social network sites. Many registrants briefly browse a MOOC, while only a smaller group persist and become more engaged over time.

The asynchronous nature of the MITx and HarvardX courses also turned out to be an important factor related to engagement: Some students registered for courses months in advance of the start date, and other registered months after the course had launched.

In some circumstances, students joined courses in the week before the final deadline for all graded materials, completing enough assessments to earn a certificate in the final week. In some courses, as many as one in five enrollments took place after the course certificate was no longer available.

Takeaway 3: Given the “massive” scale of some MOOCs, small percentages are often still large numbers of students — and signify a potentially large impact.

Demographic information about registrants can be misleading without context. The most typical course registrant in these initial courses was a male with a bachelor’s degree, age 26 or older. However, that profile describes fewer than one in three registrants (222,847, or 31 percent).

A total of 234,463 registrants (33 percent) reported a high-school education or less; 45,884 (6.3 percent) reported that they were 50 or older; and 20,745 (2.7 percent) had IP or mailing addresses from countries on the United Nations’ list of Least Developed Countries.

“While typical MOOC registrants have a college degree already, hundreds of thousands of our registrants do not. Many of our MOOC registrants are from the United States, but 72 percent are from abroad. These MOOCs are reaching many nontraditional and underserved communities of students, very different from typical students on campuses at traditional universities,” Chuang says.

One interesting subpopulation is the group of people who enroll in and complete multiple courses. More than 4,000 registrants across MITx and HarvardX earned more than one certificate, including 1,912 who earned at least one certificate from each institution. Seventy-six registrants earned five or more certificates.

“Context is so important to understanding these results,” Ho adds. “A graduate-level course in public health should attract different registrants, on average, than an introductory physics course, or a course on global poverty. In this light, it’s wonderful that so many students are engaging so deeply in courses across topics and institutions.”

Future directions

Looking ahead, Chuang and Ho view the publication of the working papers, and the open sharing of such data, as a key activity in a broad, positive dialogue in higher education.

Specifically, they emphasize that the rise of MOOCs has sparked and encouraged experimentation in teaching and in pedagogical research, benefiting both teachers and students. New tools, they contend, give faculty more flexibility and offer novel opportunities to run experiments and gather data. Likewise, online learning platforms put students in the driver’s seat, allowing an individual to engage in a manner that best suits his or her needs.

“This isn’t just about MOOCs,” Ho says. “This is about the democratization of learning: Learners are in control. We are at the beginning of an exciting effort to understand how people learn and how to educate well and effectively at scale.”

Finally, the researchers expect the working papers and data to serve as one benchmark for future studies on MOOCs and support further work in the science of learning.

“The story hidden underneath this series of reports may be this: Institutions like ours are coming to appreciate how cross-institutional educational collaborations involving many students and many courses can open new routes to understand and improve student learning — making a difference around the world and back here on campus,” Chuang says.

The lead authors of the MITx and HarvardX course reports were Daniel T. Seaton, a postdoc in MIT’s Office of Digital Learning, and Justin Reich, a HarvardX research fellow.