I recently completed my first Massively Online Online Course (MOOC), a term that presumably is at least passingly inspired by MMORGs, an online gaming genre that's most popularly represented by World of Warcraft.
The class was on Gamification and was well-taught by Wharton prof Kevin Werbach. But my focus here isn't to review or critique this particular class but, rather, to offer more general reactions to the instructional method. To reflect on what these courses seem to do well or at least handle relatively naturally, and what they struggle at. It's just a sample size of one—well, 1.5 actually as I'm currently taking a Data Science course that offers some additional insights—but I think it nonetheless exposes certain patterns.
I also encourage those interested in the topic to read Nathan Heller's "Is College Moving Online?" in the New Yorker, a thorough examination of the state of MOOCs and their potential effects on education—both for good and ill.
The format
The format for Gamification—which it seems is fairly typical—is built around a series of lectures. These consist of fairly typical Powerpoint slides with video of the instructor superimposed or off to the side in a small window. The production values are generally high and the combination of slideware and video is engaging.
There's a syllabus with links to various articles and other (free) materials. Prof. Werbach actually has a book on the topic of Gamification but it wasn't required for the course. None of the Coursera courses I've taken a look at had much if any in the way of stuff to buy.
The course then had a series of multiple-choice quizzes and a multiple-choice final exam—plus three written assignments of increasing length and scoring weight. My current Data Science course likewise has a series of lectures. But, in this case, the score comes from a series of programming and other assignments that relate to lecture topics although they are more hands-on and practical.
Type of schedule
Gamification, like the course I'm currently taking, comes from Coursera, which has a large course catalog from a wide range of schools. It was started by two Stanford professors and has received $16 million in funding from Kleiner Perkins Caufield & Byers.
Coursera's model, like that of the non-profit edX, is to offer classes in more of less real-time—by which I mean, an eight week course has a fixed start date followed by an end date about eight weeks later. Depending on the class, there may be more or less flexibility in how closely students have to hew to a weekly schedule for assignments, quizzes, and the like. But, fundamentally, the class is on a calendar and you can't dip in and out on the basis of work or family obligations, travel schedules, and inclination.
The downsides to this approach are obvious. There are a couple of classes I've considered but opted against because they overlapped periods when I would't have been able to devote much time to them.
On the other hand, after taking a class, I better appreciate why one might want to run a class to a schedule. Discussion boards, assignments (especially peer-graded ones—more on those in a bit), the availability of staff to answer course questions or address problems, and just getting forced into the "flow" of a class all require or at least greatly benefit from a schedule and associated incentive structure. (Education has a lot in common with a gamification system.)
In a related vein, I also better understand why you might not want to break a course into overly granular chunks, i.e. one or two week classes. I'm not just talking about any particular administrative overheads associated with putting a course in a catalog, but a broader set of transaction costs borne by all the participating actors such as figuring out how the class is run and understanding or dealing with prerequisites or tools. In many cases, it seems these would collectively just add too much overhead to a short course unless that course were more intensive than most people with a full-time job could undertake.
Does the fact that these Coursera courses so reflect the form and content of traditional university classes simply reflect tradition and the fact that much of the content was originally developed for such classes? Perhaps to a degree. On the other hand, whatever issues higher education may have today, it's also likely that not everything about traditional class instruction is wrong.
Another form of MOOC largely goes self-paced, even if related lectures still largely parallel the content of a semester of classes. Machine-graded quizzes can still exist, as can other types of computer- or self-evaluated assignments. Udacity, another VC-funded MOOC, follows this model today.
Other types of online instruction that increasingly diverge from a true MOOC model include iTunes University and the many instructional videos on YouTube. More focused sites, such as Code Academy, teach specific skills.
I think both more- and less-structured forms will have their place. Many of us have a limited ability to schedule courses that follow a fixed schedule and find the flexibility of watch-when-you-can attractive. At the same time, I appreciate the relative discipline and other potential benefits provided by a more formal course structure.
Grading and certification
Coursera, like edX, follows another aspect of most traditional university courses; it grades you. In Coursera's case, this takes the form of a Certificate of Completion based on your performance in various assignments and quizzes as determined by the individual class—70 percent in the case of Gamification. (Coursera is also starting to offer a "Verified" version for certain classes.)
I suspect that, at this point, a Coursera certificate is more of a gamification element, i.e. a motivator, than something that's especially useful outside of Coursera. However, it's also true that Coursera's business model will ultimately depend on being able to sell the ability to gain meaningful certifications—which means that they need to be able to grade.
Schools have, of course, been grading forever. But remember what the "M" in MOOC stands for? It's "massive," indicating that it's not at all unusual to have 50,000 people sign up for a MOOC. (Although far fewer will complete it.) It's obviously not practical for a professor and a few TAs to grade at that scale. In the future, I wouldn't be surprised to see hybrid models in which something "MOOC-like" is augmented with one-on-one and one-to-few interaction with professors and TAs for a fee—indeed we're starting to see examples of such—but let's stick to the subject at hand for today, given that what's actually being paid for starts to become a complicated question.
A couple observations about the state of grading in MOOCs.
Multiple choice works well, subject to the limitations of multiple choice. Given well-written questions, there's no more ambiguity than with any other multiple-choice exam. It's easy and instantaneously graded by computer. And modest feedback can be made available with respect to the answers.
Computers also do well at grading freeform but well-bounded and unambiguous answers, such as a numerical solution with only one correct response. It's "42" or it's wrong.
However, start talking more open-ended problems, even in quantitative topics like programming, and automated graders can start failing answers in unexpected ways. Without going into all the gory details, the autograder for the first assignment in my current data science course was highly sensitive to, for example, how the programmer chose to parse text fields and to any quirks in the output's format. While not fatal flaws, it's indicative of how automated grading challenges magnify exponentially the more creativity is allowed in solving an open-ended problem.
Also problematic is peer review. On the one hand, this offers a way for massive scale evaluation of freeform text in a way that it's hard for imagine computers tackling anytime soon. On the other hand, across a huge class with students of all ages, skills, language abilities, and interest, it's not hard to imagine that the evaluations can be… quirky—even given a relatively detailed scoring rubric.
I didn't personally have a huge problem with my results. But it was obvious from the discussion boards that a lot of people took low evaluations made without comment and evaluations made with obvious disregard for the scoring instructions very personally. And it's worth observing that, given a large class, statistics suggests that some will have simple "bad luck" with those they draw to evaluate their written assignments—even given multiple graders, multiple assignments, and algorithms to screen bad actors.
At the current stage of MOOCs, I personally find it easy enough to shrug my shoulders and get on with things. I have lots of diploma and certificate things gathering dust somewhere. But say a class has a written assignment contributing say, 33 percent of the grade. To the degree this grade has meaningful consequences for the student (for a resume or for tuition reimbursement if MOOCs start charging in some form), that's a problem. It's also fair to say that, real world significance aside, grades can have a motivating factor for many as well.
This hasn't been intended as an exhaustive look at the "grading issue" but it's been evident to me it's something that will have to be at least improved on—not that students are ever wholly happy about their grades—as MOOCs look to start collecting money from people and offering meaningful certifications.
Overall
I got a lot out of this course and it looks as if there's a lot of quality content out there—more certainly than I'll have time to dig into. I'm also happy to see both for-profit and non-profit initiatives probing at ways to make higher-education better and more efficient.
What I don't have a real opinion on is what the effect of Coursera, edX, and their ilk will be. The effect will certainly be uneven although one wonders if some aspects of MOOCs can replace elements of classes even at elite institutions. (Though I'd note that we've had the technology to replace 500 student freshman lectures for at least a decade.) I do suspect, or at least hope, that MOOCs—or at least MOOC methodologies—can replace low-value-add broadcast education in many situations.
One of the reasons that this particular crystal ball is cloudy is that higher education is often this odd hybrid of credentials, socialization, and learning that can be impersonal, highly personalized, solitary, involving lots of peer interaction, or some combination all of those. MOOCs clearly don't address all those modes but it arguably can do a subset rather well.
No comments:
Post a Comment