In March 2012, MIT launched 6.002x, a free online version of MIT’s introductory course in circuits and electronics. The course, the first massive open online course (MOOC) offered by MITx — and also the inaugural offering from edX, the online-learning partnership later founded by MIT and Harvard University — sparked worldwide interest, along with a large amount of data.
Almost 155,000 people registered for the course; throughout the semester, users clicked and scrolled through lecture videos, tutorials and discussion threads, generating more than 230 million interactions with the online platform.
Researchers from MIT and Harvard are now trying to make sense of this data, which includes students’ clickstreams (recordings of where and when users click on a page) and their homework, lab and exam scores, as well as comments made on discussion forums and responses to an end-of-course survey.
Within this data, researchers hope to find answers to some common questions about online learners, such as their demographics and how they use online resources. Data from 6.002x may also help to answer more complex questions: What factors encourage users to stick with an online course? What helps or hinders online learners’ achievement or performance?
In a paper published this month in the journal Research & Practice in Assessment, the MIT-Harvard team reports preliminary results from its analysis of 6.002x data on users’ characteristics, study habits and motivations for taking the course. The team included lead author Lori Breslow, the director of MIT’s Teaching and Learning Laboratory (TLL); physics professor David Pritchard, who heads MIT’s Research in Learning, Assessing and Tutoring Effectively (RELATE) group; and Andrew Ho, an assistant professor at the Harvard Graduate School of Education.
Pritchard, the Cecil and Ida Green Professor of Physics at MIT, says the incredibly detailed data gathered from 6.002x enables researchers to observe learning habits that might be difficult to discern in on-campus courses.
“We can study things like how much of a textbook they read, and what they said to their peers, which we can’t study on campus,” Pritchard says. “We can see everything the students do. And that’s unprecedented in studying on-campus education.”
MIT coursework around the world
Throughout the first semester of 6.002x, up to 24 computer servers recorded nearly a quarter-billion user interactions, including 12,000 discussion threads and almost 100,000 individual posts. These interactions generated 110 gigabytes — “a small-town library’s worth of data,” according to Pritchard.
The team mined data on demographics, and found that 6.002x students logged in from 194 countries, with the top five being the United States (26,333 registrants), India (13,044), the United Kingdom (8,430), Colombia (5,900) and Spain (3,684). Surprisingly, the researchers found only 622 individuals logged in from China — a far lower number than had been expected.
About 67 percent of registrants listed English as their primary language; the next largest group (16 percent) listed Spanish. The researchers noted that some students who were not native English speakers formed groups on Facebook to help each other through the course.
Toward the end of the semester, students were asked to fill out a course survey developed by TLL. Of those who responded, 27 percent had only a high-school diploma; 37 percent had completed a bachelor’s degree; and another 28 percent held a master’s or professional degree.
More than half of survey respondents cited the gain of knowledge and skills as a primary motivator for taking 6.002x, while one-quarter of users signed up for the “personal challenge,” and 8 percent were motivated by the possibility for employment or job advancement. The researchers also noted that in the course discussion forum, users with various backgrounds, from high-school students to retired electrical engineers, commented that they simply wanted to see if they could make it through an MIT course.
Talking in class
In their analysis of 6.002x resource usage, Pritchard and RELATE postdocs tallied clickstream data, such as where and when users clicked on videos, discussion threads, tutorials or textbook pages when working on homework, in comparison to when they were taking the midterm or final exam.
Interestingly, the group found that in completing homework assignments, users spent more time on video lectures more than any other resource. However, during an exam, students referred most to the online textbook, which they virtually ignored when doing homework. The data, although preliminary, illustrate how students may use different online strategies to solve homework versus exam problems.
While use of the discussion forum was not required in the course, the researchers found it to be the most popular resource for students completing homework assignments. In fact, 90 percent of the clickstream activity on the forum came from users who viewed existing threads without posting comments.
Breslow, who is also a senior lecturer at the MIT Sloan School of Management, says TLL researchers are also analyzing the content of students’ posts to see whether participation in discussions helps to improve student achievement in the course.
“We see that sometimes someone would post and say, ‘I can’t do this, it’s just too hard,’ and then other students will come on the forum and post, ‘Yes, you can, you can do it,’” Breslow says. “So even something as simple as encouragement from peers, as we know from research into traditional teaching and learning, will often help improve performance. Will it do that online in MOOC environments as well?”
From their analysis, the researchers note that students who went on to earn certificates in the course — an achievement which required scoring a total of at least 60 points in homework assignments, labs and exams — were the most active group on the discussion forum. While only 3 percent of all students took part in discussion threads, those who earned certificates were much more active: 28 percent asked a question, 41 percent answered a question, and 36 percent posted a comment.
Peer interaction seems to improve a student’s chances of success in 6.002x. While the researchers found no correlation between achievement and age or gender, they say there may be a relationship between achievement and collaboration: In particular, they found that students who reported working with another student on a problem offline tended to score almost three points higher than someone working alone.
Not surprisingly, a background in differential equations also correlated with a higher total number of points scored in the course.
Steven Mintz, a professor of history at the University of Texas at Austin and executive director of the University of Texas System’s Institute for Transformational Learning, says the new research establishes a noteworthy link between a student’s social interactions and her success in an online course.
“Educational success in a MOOC, but also in a face-to-face class, is not a wholly individual activity,” says Mintz, who was not involved in this study. “It has a social dimension. To put this another way, persistence and success are not simply products of cognitive factors. Noncognitive factors — in this case, social connection — are equally important.”
Dropout rating
One of the many questions TLL researchers plan to tackle as they wade through the course data is what determines whether a student sticks with the course. In 6.002x’s first semester, of the almost 155,000 who registered, only about 7,000 received certificates — a precipitous drop, at first glance.
But Pritchard says a closer look at the data reveals a subtler picture of online learning. By analyzing student activity in stages — from registration through the first and second homework assignments to the midterm and final exams — he found that two-thirds of those who registered dropped off almost immediately, signing up only to never return.
More interestingly, he found that of those who stuck with the course through the second homework assignment, 40 percent went on to earn a certificate.
“That’s not so different from an on-campus course,” Pritchard says. “Students often sign up for several electives, then drop one.”
A question emerging from the data is whether 6.002x and other online courses can incorporate strategies to help retain students who kept up with most of the course, but failed to earn a certificate.
“Does this mean that edX courses should be engineered in a different way if you’re trying to keep people in the course, as opposed to trying to get people to achieve at a certain level? We simply don’t know, and it bears more research,” Breslow says. “We’re forging new territory in terms of how you work with this data. We’ve never had anything in educational research like this.”
This research was supported by the National Science Foundation.
Almost 155,000 people registered for the course; throughout the semester, users clicked and scrolled through lecture videos, tutorials and discussion threads, generating more than 230 million interactions with the online platform.
Researchers from MIT and Harvard are now trying to make sense of this data, which includes students’ clickstreams (recordings of where and when users click on a page) and their homework, lab and exam scores, as well as comments made on discussion forums and responses to an end-of-course survey.
Within this data, researchers hope to find answers to some common questions about online learners, such as their demographics and how they use online resources. Data from 6.002x may also help to answer more complex questions: What factors encourage users to stick with an online course? What helps or hinders online learners’ achievement or performance?
In a paper published this month in the journal Research & Practice in Assessment, the MIT-Harvard team reports preliminary results from its analysis of 6.002x data on users’ characteristics, study habits and motivations for taking the course. The team included lead author Lori Breslow, the director of MIT’s Teaching and Learning Laboratory (TLL); physics professor David Pritchard, who heads MIT’s Research in Learning, Assessing and Tutoring Effectively (RELATE) group; and Andrew Ho, an assistant professor at the Harvard Graduate School of Education.
Pritchard, the Cecil and Ida Green Professor of Physics at MIT, says the incredibly detailed data gathered from 6.002x enables researchers to observe learning habits that might be difficult to discern in on-campus courses.
“We can study things like how much of a textbook they read, and what they said to their peers, which we can’t study on campus,” Pritchard says. “We can see everything the students do. And that’s unprecedented in studying on-campus education.”
MIT coursework around the world
Throughout the first semester of 6.002x, up to 24 computer servers recorded nearly a quarter-billion user interactions, including 12,000 discussion threads and almost 100,000 individual posts. These interactions generated 110 gigabytes — “a small-town library’s worth of data,” according to Pritchard.
The team mined data on demographics, and found that 6.002x students logged in from 194 countries, with the top five being the United States (26,333 registrants), India (13,044), the United Kingdom (8,430), Colombia (5,900) and Spain (3,684). Surprisingly, the researchers found only 622 individuals logged in from China — a far lower number than had been expected.
About 67 percent of registrants listed English as their primary language; the next largest group (16 percent) listed Spanish. The researchers noted that some students who were not native English speakers formed groups on Facebook to help each other through the course.
Toward the end of the semester, students were asked to fill out a course survey developed by TLL. Of those who responded, 27 percent had only a high-school diploma; 37 percent had completed a bachelor’s degree; and another 28 percent held a master’s or professional degree.
More than half of survey respondents cited the gain of knowledge and skills as a primary motivator for taking 6.002x, while one-quarter of users signed up for the “personal challenge,” and 8 percent were motivated by the possibility for employment or job advancement. The researchers also noted that in the course discussion forum, users with various backgrounds, from high-school students to retired electrical engineers, commented that they simply wanted to see if they could make it through an MIT course.
Talking in class
In their analysis of 6.002x resource usage, Pritchard and RELATE postdocs tallied clickstream data, such as where and when users clicked on videos, discussion threads, tutorials or textbook pages when working on homework, in comparison to when they were taking the midterm or final exam.
Interestingly, the group found that in completing homework assignments, users spent more time on video lectures more than any other resource. However, during an exam, students referred most to the online textbook, which they virtually ignored when doing homework. The data, although preliminary, illustrate how students may use different online strategies to solve homework versus exam problems.
While use of the discussion forum was not required in the course, the researchers found it to be the most popular resource for students completing homework assignments. In fact, 90 percent of the clickstream activity on the forum came from users who viewed existing threads without posting comments.
Breslow, who is also a senior lecturer at the MIT Sloan School of Management, says TLL researchers are also analyzing the content of students’ posts to see whether participation in discussions helps to improve student achievement in the course.
“We see that sometimes someone would post and say, ‘I can’t do this, it’s just too hard,’ and then other students will come on the forum and post, ‘Yes, you can, you can do it,’” Breslow says. “So even something as simple as encouragement from peers, as we know from research into traditional teaching and learning, will often help improve performance. Will it do that online in MOOC environments as well?”
From their analysis, the researchers note that students who went on to earn certificates in the course — an achievement which required scoring a total of at least 60 points in homework assignments, labs and exams — were the most active group on the discussion forum. While only 3 percent of all students took part in discussion threads, those who earned certificates were much more active: 28 percent asked a question, 41 percent answered a question, and 36 percent posted a comment.
Peer interaction seems to improve a student’s chances of success in 6.002x. While the researchers found no correlation between achievement and age or gender, they say there may be a relationship between achievement and collaboration: In particular, they found that students who reported working with another student on a problem offline tended to score almost three points higher than someone working alone.
Not surprisingly, a background in differential equations also correlated with a higher total number of points scored in the course.
Steven Mintz, a professor of history at the University of Texas at Austin and executive director of the University of Texas System’s Institute for Transformational Learning, says the new research establishes a noteworthy link between a student’s social interactions and her success in an online course.
“Educational success in a MOOC, but also in a face-to-face class, is not a wholly individual activity,” says Mintz, who was not involved in this study. “It has a social dimension. To put this another way, persistence and success are not simply products of cognitive factors. Noncognitive factors — in this case, social connection — are equally important.”
Dropout rating
One of the many questions TLL researchers plan to tackle as they wade through the course data is what determines whether a student sticks with the course. In 6.002x’s first semester, of the almost 155,000 who registered, only about 7,000 received certificates — a precipitous drop, at first glance.
But Pritchard says a closer look at the data reveals a subtler picture of online learning. By analyzing student activity in stages — from registration through the first and second homework assignments to the midterm and final exams — he found that two-thirds of those who registered dropped off almost immediately, signing up only to never return.
More interestingly, he found that of those who stuck with the course through the second homework assignment, 40 percent went on to earn a certificate.
“That’s not so different from an on-campus course,” Pritchard says. “Students often sign up for several electives, then drop one.”
A question emerging from the data is whether 6.002x and other online courses can incorporate strategies to help retain students who kept up with most of the course, but failed to earn a certificate.
“Does this mean that edX courses should be engineered in a different way if you’re trying to keep people in the course, as opposed to trying to get people to achieve at a certain level? We simply don’t know, and it bears more research,” Breslow says. “We’re forging new territory in terms of how you work with this data. We’ve never had anything in educational research like this.”
This research was supported by the National Science Foundation.
