In a series of shocking studies, learning environments capable of tracking trillions of learner actions down to the millisecond have led to a breakthrough in educational research: being active in a learning environment can be a powerful indicator of success in this learning environment. That is, students who do stuff also do more stuff, and they do stuff better.
This powerful idea, that doing things may very well be one of the reasons people do more things or do good things, is beginning to come into focus after a series of studies looking at some of the biggest online learning platforms, such as Khan Academy, Google Course Builder, Udacity and edX.
One of the promises of big data in education is that online learning platforms that collect learner actions in real time across thousands or millions of learners will lead to major advances in teaching research. and learning. edX President Anant Agarwal called edX a particle accelerator for learning. So let’s see what happens when we start crushing students at extremely high speeds.
Let’s first look at the recent study the implementation of Khan Academy in 20 schools, led by SRI International and funded by the Bill and Melinda Gates Foundation. SRI spent two years studying KA in schools from 2011 to 2013, and their methods were diverse; in their own words: “To gather information on how Khan Academy was being used and its potential benefits, SRI researchers visited schools, districts, and CMOs; makes observations in class; interviewed leaders of organizations and schools as well as teachers, parents and students; conducted teacher and student surveys; and analyzed student user log files during the school year. Huge study, multiple methods, but let’s focus on big data: logs.
Khan Academy’s log files contain detailed data on students’ behavioral activity: how long they watched a video, how many times they solved a problem, etc. SRI researchers took this vast wealth of data and boiled it down to a single statistic: the number of minutes a student logged into Khan Academy.
This is a development we should expect to see quite often in future research: where researchers take big data and turn it into small data. It’s actually quite complicated to look at all those log files and user activity pathways. So researchers are taking terabytes of data, putting all of those nuances and complexities together, and boiling down the big data into a single number.
And the big discovery here was this: there is a correlation between the number of minutes a student spends on Khan Academy and test scores. Here is the key chart, note that students who do better than expected spent more minutes on KA:
The SRI study is careful to note that this is not incidental evidence. It could be that kids who spend more time on KA also listen more in class. It could be that kids who do well in math like CA more and spend more time on it. But here’s the first piece of our puzzle: Kids who do more math at Khan Academy do better on math tests.
Udacity and San Jose State University
Khan Academy is laying the groundwork, but let’s move on to higher education. In the summer of 2013, San Jose State University partnered with Udacity to offer a series of online remedial and introductory courses. This fall, they presented some preliminary results in a research report. Once again, Udacity collects detailed, real-time clickstream data showing every action every student takes on the platform. Big Data. Huge data. So, once again, the researchers compressed this rich, detailed data into very simple summary statistics, such as number of issues resolved and minutes of video viewed. And, again, the conclusions are provocative. It turns out that students who got things done were more likely to pass the class. In a convoluted and audacious seeker, speak:
The main finding of the model, in terms of importance to course success, is that measures of student effort trump all other variables examined in the study, including student demographic descriptions, course topic and student use of support services. While support services may be important, they are overshadowed in current models by the amount of effort students put into their courses. This overall result may indicate that responsible student activity—problem sets, for example—may be a key ingredient for student success in this environment.
Researchers here are cautious. Again, this is not a causal study, like a randomized controlled trial. But evidence suggests that responsible activity – for example effort, for example doing things – *can* be a key ingredient for student success, for example passing a class.
Here, the data is presented in graphical form. In the first figure, we see that the probability of passing a course is strongly correlated with the problems of doing. Watch these lines go up.
In the second figure, we see that hours of video watching are also correlated with course completion, especially in this one class of statistics.
Once again we see that by taking big data, reducing it to simple summary statistics and creating basic statistical models, we find evidence that effort predicts success, that getting things done can be correlated with doing good in those same things.
Google Course Builder
Next, let’s move on to Google’s Mapping with Google data Classes. Google has one of the largest teams of top data scientists in the world, so I think it would be fair for readers to come into this section with real excitement about what might be unfolding.
A team of three Google researchers recently submitted a paper to the [email protected] conference, and some of their conclusions build on this growing evidence base that doing things leads to doing things. Here Google researchers claim that “our research indicates that learners who complete activities are more likely to complete the course than their peers who did not complete any activities” (h/t at Hack Education)
To defend this controversial and still unproven position, the researchers compared students who did and did not do activities during the course with those who completed the final project. Again, Google has real-time analytics on every action taken by every student in the course and one of the largest teams of analysts in the world, and they’ve chosen to whittle this massive trove of data down to simple statistics. summaries. In fact, they get particular praise for reducing things to dichotomous variables like whether or not a student did something, rather than more complex variables like how many times they did something or how well they have done it.
The evidence here, presented in both a table and a bar chart for emphasis, shows strong evidence that students who did the activities during class were significantly more likely to do the final project than those who did not. didn’t do the activities.
The table shows it. The bar chart shows it. So it might be fair to consider that this study provides twice as much evidence as previous ones that doing things seems to be correlated with doing more things.
My own contribution to Do Stuff and Do Stuff Theory
So that’s the proof of Khan, Google and Udacity, but now it’s time to get into the particle accelerator itself: edX. I’ve personally conducted several studies on HarvardX courses, and I’m ready to add my bricks to the pile of evidence presented so far. By now the script should be completely familiar. We have gigabytes of log data on student activity in our courses, and with MITx, we have a team of three post-docs working on this research, with access to professors of computer science and education from foreground to support us.
As in the previous cases, in the reports of which I was the main author (and here I will start using the first person singular so as not to implicate my colleagues in this nonsense), my first gesture was to reduce this Incredibly rich content and nuanced data in a single measure of student effort. And what better way to summarize clickstream data than to simply count the number of “clicks” or log events for each student. Then I don’t even bother with patterns or correlations, I just present histograms of the number of events for those who pass and don’t pass the course. I’m sure at this point you can predict the results. In several different courses, I’ve found that people who passed a course and earned a certificate, on average, clicked on things more than those who didn’t. Here are the Justice and Heroes miniatures:
As my colleagues and I wrote: “In general, however, those who took many actions in the course were more likely to earn a certificate than those who took few actions, a simplistic idea that echoes to the findings of other previous studies on online courses based on lectures.
Harvard research confirms it: people who pass the courses do more things than the others.
Reich’s Law of Things to Do
At this point, the patterns of so many courses, in so many disciplines – humanities, sciences, mathematics, philosophy – all pointing in the same direction can only lead to one conclusion. Thus, I am ready to risk my reputation by proposing a new scientific law for research learning. I propose Reich’s Law of Things to Do: students who do things in a MOOC or other online learning environment will, on average, do more things than those who do nothing, and students who do things will perform better than those who do nothing . The implication for practice could not be clearer. When we create online courses, if we want students to learn, we need to motivate students to actually do things in those courses, rather than getting bored and going off to watch cat videos (unless the MOOC in question is something like, The Semiotics of Viral Cat Videos).
In fact, I would argue that this law is so well established that we may not need additional published studies that demonstrate this point. This could perhaps be a corollary of Reich’s Law: if a study of e-learning simply shows that activity predicts extra activity or effort, we may not need to spend time to write this study.
Two questions for further research are: With all the extraordinary data resources that are available to MOOC researchers, why do so many people reduce big data to small data? And why does a major body of research appear to confirm the obvious? I will save this discussion for a future article.
For regular updates, follow me on Twitter at @bjfr and for my publications, resume, and online portfolio, visit EdTechResearcher.