Category Archives: Higher Education

Posts related to higher education

On MOOCs and Accreditation

Recently Udacity, one of the major MOOC providers, announced that, in a shift in focus, it is eliminating certificates of completion for non-paying students. The free students can still look at materials and take part in activities, but if they want anything to show for it, the only option is to pay $150 a month as a verified student. The basic pdf certificates of completion that used to be made available to anyone, paid or free, that worked through the material and passed, are being phased out.

This is a mistake, a move 180 degrees in the wrong direction in the ongoing development process of a model for MOOCs and accreditation. In reality, not only should the free students be able to keep gaining some sort of certification in recognition of their efforts, but further moves need to be made to start accrediting these online courses with the universities giving the courses. MOOC consortiums actually need to be leaning harder on schools to make meaningful accreditation available to students rather than taking away elements they were already offering.

Now, don’t get me wrong; the new features Udacity is rolling out to those students paying the extra money are great steps in the right direction- added human support, and grading of submitted work by professors and others with subject expertise rather than peer grading with rubrics. This is great stuff, and if it helps the resulting certificates get taken more seriously in the marketplace, that is great. All of this helps to further the legitimacy of the online medium as a way to get university level education and continuing professional development.

But that’s no reason to throw the free students under the bus. Fine, Don’t let the free students be able to get the assignments graded by profs and TAs. This is expensive, and if someone doesn’t pay for it, it’s probably an unsustainable model, given all the other costs of putting on a nice MOOC.

Let the free students stay graded by peers. But offer them the ability to verify their identity with their webcam and a typing sample, as with the Signature Series on Coursera. (Note: the enhanced Signature Series certificates on Coursera are only available to paying customers, but the prices – $50 a course for courses that can take up to a few months, are much more accessible than the Udacity prices of $150 per month)

The technology for this sort of verification is already developed, so it doesn’t really cost the provider anything extra. And then give the free students a certificate for their efforts. Maybe that certificate is going to have a second rate market value compared to the paid certificates based on professor or TA grading. But at least they will have something of value to take away.

Yes, I realize that the paying students are the ones that pay the bills and keep the lights on. But MOOCs are a big deal in the first place because of the mass of free students. That is what is driving the traffic, the buzz. These millions of students are freely choosing to spend their free time in higher studies rather than vegging in front of the TV. Educational institutions should love this, and should want to encourage it.

The fact of the matter is that MOOCs blew up as an idea because of a promise, explicit and implicit, made to people. MOOC consortiums actively sold the idea that people could not only expand their knowledge, but also expand their opportunities through hard work in these online courses. The barriers to access to higher education would be lowered. If these masses of students lose faith in that vision, the whole thing will collapse. The numbers will shrink. With smaller overall enrolment, there will be correspondingly fewer that will stick around to pay for it. Part of that mass of free students eventually takes the plunge to pay for some sort of extras like an enhanced certification or a proctored examination. The free education generates paid education by helping to support a vibrant platform. The numbers also generate buzz and perceived legitimacy of the educational platform as a place to get meaningful education and training. This perceived legitimacy is key to attracting people to spend actual money on a course.

If you let the less well off students get certification and / or credit if they are willing to (1) do the work and (2) verify their identity for exams and assignments, they will be able to use that credit in professionally meaningful ways to help get into fruitful careers. Given a fruitful career path, these students can have the means to later become paying students in the future as they continue their professional development.

This is the sort of social welfare outcome that people in higher education say they want, right? Then make it happen. This should be the sort of thing both Liberal-minded (give to help the less fortunate) and Conservative-minded (remove barriers to let people raise themselves through their own determined effort) academics and administrators should be able to get behind.

Yes, it is the paid students that keep the bills paid, but understand that people are going to pay for it. If there is meaningful, professionally usable certification and credit available, working professionals will pay for it. It’s like with digital file sharing. It still exists, and is booming, but with reasonable and convenient means available to get access online, people pay for the content.

And if you demonstrate social responsibility by making perhaps lesser but still meaningful and usable credentials to those who can’t afford it, then those who can pay will be even more driven to support you. Because you will have proved your worth as an organization to get behind.

Peer Assessment in MOOCs


As mentioned in some previous posts, I’m currently taking a MOOC (Massive Open Online Course) through Coursera. I’m overall very pleased with the course, enjoying it a lot, learning a lot. It’s my second MOOC course, and the first I’ve taken from one of the big three vendors (Coursera, Udacity, and EdX). In the process, I’m getting a lot of great insights in terms of thinking about design for this exciting new format of university education.

One area where I am pleasantly surprised is in how well the aspect of peer assessment in MOOCs works for assignments. For those who are not so familiar with the format, in the MOOC environment, there can be thousands of students per course (this is where the “Massive” part comes in). As a result, instructor or facilitator grading is not practical or cost effective. Now, for some types of assessments, such as multiple choice or short answer or matching or what not, these can be graded by machine. The challenge is grading of more open-ended work like essays, projects, problem sets, and what not. These more complicated types of submissions cannot readily be done by a machine.

As an alternative, therefore, “peer assessment” is used. That is, the grading is carried out by the students themselves.

Peer Assessment in MOOCs: How it Works

Each student gets randomly assigned five other student submissions (anonymous) to grade using an online grading form. A rubric guides the marker on areas to assess, and criteria for assigning or subtracting marks.

Here’s an example of a rubric used for a recent assignment on programming a simple virtual stopwatch (with the stopwatch itself on the right):

stopwatch_rubric  stopwatch


A web form based on the rubric is used to collect the assessment feedback. Drop down menus are used to select grades on each rubric item (0/2, 1/2, 2/2, e.g.) with criteria given for each possible mark for each item. Text boxes are given for more detailed feedback, to give the student some explanation when marks are taken off or to give some positive feedback for a truly impressive submission.


The rubrics help to focus the markers and to standardize the grading for improvements in the all-important Inter-Rater Reliability. For people to take MOOCs seriously, there needs to be that sense among students that the grading is as uniform and objective as whatever grading they would get in a normal university course. (I won’t get into the thorny question of how much that exists in current brick and mortar university classrooms 😉 ) Assigning multiple submissions to be marked by each student also means that each student will in turn get assessed by multiple peers. This adds another level of objectivity on top of the rubric; variance between markers can be washed out by dropping outlier grades (lowest and highest) and averaging over the remaining grades.

Overall this encourages objective, fair, and thorough grading. Extremely lax/superficial grading or abusive, over-strict grading tends to get dropped out when the low and high outliers are dropped. Also, there is a bit of a “do unto others” / karma effect here. The person you are grading this assignment could well be grading you the next assignment, so you don’t want to be abusive of the power. On the other hand, you know others are going to be marking down your errors if you made them, so you wouldn’t want to be over-lenient.

The student duty to contribute to marking is reinforced by making it part of the criteria for success; students lose grades on the next assignment if they fail to grade their minimum of five assignments. The option is also given to choose to mark more than the minimum. In the course I am taking, this is voluntary, and does not give any added benefit. I would probably suggest to the makers of the course or Coursera that some additional marks or offsets for errors in other assignments might be one way to incentivize learners in the desired behavior of going beyond the minimum.

Another neat touch is self-assessment. After grading the requisite five peers, the student is asked to evaluate his own work. This also becomes part of the grade. From a perspective of Adult Learning Theory, this gives a nice opportunity for the learners to reflect on their own work after a few days away from it, and in the light of what he has seen of the mistakes and highlights of what others have done, helping to further reinforce learning.

What we can learn from this in general

Overall, this is a system that works nicely, and helps to reinforce the idea that MOOCs can work in practice to replicate a university course online.

I think, however, we could learn some takeaway lessons from this format for our design of e-Learning in general. Assessment can often be a weak point in e-Learning / m-Learning due to the limitations in what can be graded by machine. This is why a lot of the assessment in e-Learning is in the form of simple, closed answer forms readily supported by authoring tools, things like True/False, multiple choice, short answer, drag and drop, and matching. Some interesting assessment options can be engineered for soft skills training by combining multiple choice questions with slide branching to create “Choose Your Own Adventure” style interactive scenarios. But open ended question answer types like essays, video responses, or project submissions are not so readily supported.

Perhaps we can take some inspiration from this technological implementation of peer assessment and self-assessment in MOOCs to enable simple grading of more complex assessments in e-Learning. At least for cases where we have a large number of students taking the course at the same time.

All that would be required from an Instructional Designer would be the design of easy to use and understand grading rubrics. In addition, the Instructional Designer would need to design web forms for the students to enter marks with drop down menus for each rubric item, and to enter open ended grader comments.

To handle all the magic the background, programmers would need to implement an engine for:

  • Receiving submissions and enforcing any soft/hard deadlines
  • Randomly assigning submissions to students for grading
  • Receiving and processing the submitted grades and calculating final assignment grades
  • Communicating the results to the learners

In summary, online peer and self-assessments based on rubrics and supported by the right IT functionality offer refreshing new possibilities for richer forms of assessment in e-Learning, given a large enough pool of students taking the course at the same time.

Observations on MOOCs and Teaching Styles

I’ve been spending some time in the past year taking MOOC courses. MOOCs, for those not familiar with this recent buzzword in education, are Massive Open Online Courses. They are courses built by universities, taught by university professors, but delivered through the internet to anyone who wants to take them, anywhere. For anyone who just wants to learn the material and go through the exercises, the courses are usually free to take. There are, increasingly, options to pay for the chance to take an exam for university credit or to get a certificate of completion. But if you just want to learn about some subject that you’re curious about at a university level, it is a great option.

University MOOCs are delivered, for the most part, through large consortiums such as Coursera, Udacity, and EdX, but the individual courses themselves are built by the universities themselves, and the universities maintain their own homepages through the consortium sites and maintain some university branding.

One course I took last summer was on Teaching with Moodle, put on by the folks at Moodle. Another one I’m taking now is from Rice University through Coursera on Introduction to Interaction Programming in Python.

This has been an illuminating experience. One of those things that is on my wishlist in terms of projects I would like to work on professionally is to work as an Instructional Designer helping university professors to make MOOCs. And over the past year as a student I’ve learned a few things. I hope to publish a longer post detailing all my observations once I’ve taken a few more courses and seen more variety in terms of how the concept is implemented well, and how it is implemented poorly.

I’d like to take the time now to share a couple of these early observations.

The first key observation is about the general need for buy-in from classroom instructors if you are either relying on them as subject matter experts (SMEs) for self-paced e-Learning or helping them to teach using an online platform, whether through asynchronous pre-recorded videos or a virtual classroom. They are a key stakeholder, and if they’re not comfortably on board, the process is going to be much harder and the results less successful than they could have been. This is the sort of thing they teach you in Educational Technology courses, but the understanding of this became much more concrete and visceral in practice. I learned this basic lesson well earlier in my career while working on a 15 month long project with a handful of courseware developers focusing on converting certain theoretical material in an Instructor Led Training course on maintaining Full Flight Simulators to self-paced e-Learning. This involved working with a number of instructors. One of the big challenges in working with the instructors was getting their full engagement and confidence. Some of them saw it as some sort of potential challenge to their livelihoods. Which is a hard mindset to deal with if you want to encourage them to give of their time.

The second observation is more specifically for Instructional Designers who would find themselves as Online Pedagogical Consultants in universities building MOOC content or helping instructors do so.  We want instructors to be comfortable with MOOCs and not them not as a threat. Rather, we want them to see it as just another media through which they can express themselves. To do this, though, you need to give them space to express their personality in the style of presentation.

The programming course I am taking gave a good chance to see this, as it uses four different Rice University Computer Science professors. The content presentation videos were short, ranging from about five minutes up to a max of about 15 minutes, and these videos were distributed between the different professors. Each of them has their own personal style of presentation. Their individual personalities come through. And as a learner, it was nice. I prefer some of them over others, but nevertheless, I like that it gives some variety. It makes it feel like the mix you see in a university.

This may be a challenge for some of us as Instructional Designers, as our usual instinct in online training is to try to find ways to standardize stylistic aspects of communication across courses. This is usually because we feel some pressure from marketing types with a stake in the programs, wanting to “establish a consistent brand.” There is something to that, and there is an element there where we have to standardize look and feel so that students can get a predictably good experience. This includes the look and controls of video/media players and the overall visual “container” template, e.g. MOOC delivery platform logo and login information, school logo, where the course title is, and where the high level navigation elements of the course are on the page. These are essential visual navigation and control elements that need to be standardized for better learner experience. As well, we do need to take the advantage of our time with the professors to give whatever coaching we can to impart some known tips for better lesson delivery, whether on video or in the classroom. But the point is that it can’t be too homogenized.

There is a value to the students, qualitatively and likely quantitatively too in allowing the professors to express their unique character. It brings the course to life more like a real classroom and makes it much less industrial and sterile. (Which may do something to help with student motivation to complete and help improve the dismal completion rates for MOOCs) There is perhaps sometimes a place for sterile, industrialized training, but I don’t think the university should give up the ghost just yet.