Formative Assessment: Why should students care?

Formative assessment is, in a nutshell, assessment that helps to guide your learning in the future, rather than measuring what you learned in the past. It guides students on where they might need to focus their study and teachers on where they might need to review or cover topics in more details. Online, machine gradable assessment tools have big practical advantages in this area. The instant grade means instant feedback to students - not two weeks later when the module is nearing it's end. An online, machine gradable approach means formative assessment can be 'small and often', without crushing the lecturer under a paper mountain.

One of the two great challenges of online formative assessment is how to actually get the students to do it if it does not contribute directly to a final grade. Assessment drives behaviour. Students will do almost anything (bar study!) to get grades. The rational utilitarian human, beloved of economists, fills our campus, if not our lecture halls. A cunningly devised 1000 question MCQ bank might be a great tool for them to assess their level of knowledge, but if there is no grade attached to it, and they know the final exam is two essay questions, it'll never see a click. They may agree with their lecturers that the formative assessment will help them learn, and be of great benefit, but they won't do it.

Medical Students, of course, are seldom far from their MCQ practice books, always testing their knowledge against practice questions - self directed formative assessment. This is because so much of their assessment appears to rest on questions of this type (if not here in UCC, beyond it). The practice questions are a valid simulation of the real assessment event - they have authenticity to the students. That doesn't mean you need to turn your final exam into an MCQ fest. Questiosn about the content of an audio file might be great prep for an oral exam in languages. Questions about an image could be useful prep for a practical exam in mineralogy or pathology.

Another approach is to get them to do the formative tests in class. This is tricky for an online assessment, as they may not have easy web access in class. If you revert to paper, you are back to marking again (unless you take the time in class to do a peer marking exercise).

Feedback is also important. If the students know that you will take note of the results to guide your teaching - revising topics where students scored poorly, they are more likely to pay attention to the assessments as worthwhile expenditure of their precious time.

Bear in mind is that machine gradable questions are not all MCQs - 'Who want's to be a millionairre' style fact recall tests. Blackboard alone supports 14 different question types. With creativity and thought, they can be genuine tests of high level reasoning an critical thinking. I'm currently collating an example set of such 'smart questions' for a workshop on the topic (contributions welcome!).

There is also no reason you can't mix in non-machine gradable questions in your online assessment. Most systems will support questions with free text responses, and even two or three in the mix could lift the test to a higher level. For example, in the humanities, you might ask some MCQ style questions about whether a given historical text (or audio of a speech) is incomplete, biased or unbalanced in one direction or other to test a readers critical reading skills. Then have a short free text question where they elaborate on why they imagine the bias exists. It will slow down the assessment, but not as much as running full pen and ink test.

The Centrality of Outcomes

On Monday, I attended a workshop given by Prof. Harry Hubble (more details here) on Curriculum Development. Harry is well respected in the International Teaching and Learning community, and had a great deal of value to give, substantially more than could be delivered in a two hour workshop. The workshop was captured, and will be available on the NAIRTL website on video in due course.

What I picked up from the workshop was the centrality of having clearly and well defined learning outcomes before setting out to design your curriculum. If you don't have clear definition and agreement on what your course/programme/institution is trying to achieve, good luck with getting any coherent output.

One of the in-class exercises Harry set was to come up with learning outcomes at an Institutional level. What are the Universal Learning Outcomes one of our graduates (regardless of discipline) should achieve?

People came up with lots of good ones, to my mind, they all fitted into a framework of three pillars:

The Craft of Learning

They must have learned how to learn. They are capable of self directed learning. They are committed to lifelong learning and continious development. This is vital in a world where the half life of technical knowledge is increasingly short.

The Application of Reason

They must be able to do high order critical thinking, to Reason, to see past the headline. They must have the information literacy to assess the quality of evidence, the capacity to develop a reasoned course of action based on that, and an understanding of the scientific method (in the broad sense) so they can adapt their strategies in light of what works.

The Skills of the Discipline

They must an acceptible competance in the skills of their discipline. Calculate a dosage, conduct an interview, design a doorway - they actually have to be able to do what the degree says they can do.

I had it all wrapped up in this neat framework until somebody mentioned Ethics, which didn't even occur to me. I'm a child of the 80's. We don't do Ethics.

Nevertheless, the framework was useful to me in understanding what a University should do, and how important the first two pillars are. Universities claim to operate above the level of Technical Colleges and vocationally focused tertiary education. If we don't deliver on those first two pillars, we are kidding ourselves.

Note:

This is slightly off-topic for the blog, but framing good outcomes at course level is going to be central, and should drive the how, which and whether of your usage of Technology Assisted Learning.

I will add the link to the Video on the NAIRTL website once it becomes available.

Audio Recording Tips

Over at The Rapid E-Learning Blog there is an excellent post with some tips on how to get good quality audio. It's really well worth a read. Many of the points, like recording 10 seconds of ambient noise, are really useful - I'm sorry to say I learned them the hard way.

Just about the only things I would add to that post are:

1: Script, but don't get too hung up on it. Reading a word for word script sounds wooden, unless you are a good actor. Working from bullit points, as you would in a lecture, is fine.

2: Always always always start the recording by stating the date, and what the recording is. We're used to text documents where we can easily open and skim the document to see what it is about. You can't really skim an audio file, and there are few more annoying things than having a large audio file, say, of a lecture, and not really remembering which one it is. This is particularly a problem when working with other people's audio, when you may not be familiar with the material.

Fast Podcasting

In the last post, I linked to Bud Hunts Podcast "Why Technology". It's worth revisiting, not only because of what he says, but of how he says it, and where.

"..coming to you from the off ramp, near my home.." says Bud. He has recorded it while driving home, on a mobile, non specialised recorder (probably an iPhone or similar). There is road noise. There are pauses. It is technically imperfect, overlong, sure. We can pick holes in it, if we like. Let's not.

What it really has going for it though, is that it exists, which beats the socks off most of the great podcast ideas in my head, and yours.

He isn't hung on on quality. There is no jingle at the start, he hasn't done a noise removal. There is no fancy recorder or expensive microphone. The segment is thought through, but not scripted. Things could be done to improve it, but the key thing is that it's good enough to communicate the message effectively.

He hasn't gotten stuck into technicalities of podcasting distribution. The link on the web is just a regular link to a html file, no different from how you would link to a word document or a powerpoint file. Forget about RSS, iTunes, streaming servers, and all the paraphanalia of formal podcasts. He just put the audio file up there, where you can get it, play it, download it, hear it.

Quality, in the end, is not an absolute thing. Quality is fitness for purpose. Don't let technical factors be an excuse for you in creating material. Just get the minimum you need and do it. That's a computer, a microphone, and someplace on the web or Blackboard to put the files. Don't let time be an excuse - Bud recorded his file on the drive home. Buy a cheap clip on mike and it's hand free. There is no legitimate excuse here.

We are, in the end, what we create and produce. The greatness in our minds (such as it is) is of no use to anyone unless we can get it out there. Tools like this quick podcast are a fast means of doing that, and capturing valuable content for your students. You have all summer long to capture some for next term. Get to it!

Twitter: Yes, but, how can you teach with that?

Twitter is the top teaching tool of 2009, according to Jane Hart's "Top 100 eLearning tools of 2009" list, available in draft on her website. Powerpoint languishes at number 9. The list is drawn from a poll of learning technologists, and is classic example of what Bud Hunt, and others, call Bright Shiny Object Syndrome - a passion for the newest, shiniest tool as a panacea for all ills.

[If, like many people, you have only just heard of Twitter and don't have a sense of what it does, I'll again direct you to Commoncraft - Twitter in Plain English]

In terms of technology adoption, I'm unrepentantly in the early majority. I like to see others prove the technology and find a usage model that works, before leaping in to try it or advocate it myself. Twitter, for teaching, isn't at that stage yet. I haven't seen a usage model for Twitter as a teaching tool that is compelling and makes use of what Twitter can do, over an above the effect that any new trick or tool might have to perk up a classes interest for a while.

There are compelling uses for Twitter beyond the purely social. I use it as a straight information dissemination channel (@ionadbairre) for events at work and others, for example @GrahamAttwell use it well as tool to spread their ideas, a personal marketing tool in effect (and nothing wrong with that). There is an interesting application for it as a sort of topic themed chat facility. For example, people tweeting from the recent EdTech conference tagged their tweets with a #EdTech09 keyword, so by searching on that you could get a sense of the buzz of activity about the conference. There is real value there. Some people use this dynamically in a presentation as a discussion or conference backchannel.

In teaching a compelling usage mode has not yet emerged, and I have my doubts if it will. There are some good efforts out there on the web. David Parry, of University of Texas, has a nice video of his usage of Twitter to connect to his students in the Chronicle of Higher Education. He makes the case for using it as a tool to keep engaged with his students, rather than a teaching tool in a strict sense. French Today, a language learning website, makes use of it as forum for discussion of vocabulary. This presentation by Tom Barrett "Twenty Five interesting ways to use Twitter in the classroom" summarises some other approaches, as does Steve Wheeler of the University of Plymouth on his blog. Steve has some great links to other resources on teaching with Twitter. Some of them have potential, such as the 'Lingua Tweeta' approach for languages.

Another place Twitter might come into play in in contexts where phones are abundant but computers and broadband are scarce. I suspect the compelling usage models of Twitter in teaching won't come from a JISC funded study, but from a school in Nairobi, or Lagos, where mobile phone penetration is going to be orders of higher than networks PCs for some time. Through it's SMS linkages, Twitter could allow them to do things we might do with a heavier tool like our Virtual Learning Environment or eMail systems.

One other barrier to consider for adoption of Twitter in a core teaching activity is you need to get everyone in class using it. David Parry notes that only half his students use it. Shifting everyone on to Twitter to carry out any kind of core teaching exercise is going to be difficult. You would need a really compelling reason to try. My advice on Twitter is, sure, check it out if you have the time by all means, but I don't think it is ready for your classroom yet.

Prove me wrong!

Video Formats made a bit simpler.

Leigh Blackhall (http://leighblackall.blogspot.com) blogs usefully about Video formats (and many other things besides). For Audio, mp3 has been the obvious option for years, but I always found Video quite confusing. Leigh does a better job than I could on explaining the main formats you should use. The only thing I would add to his post is to note the general drift towards Apple and iTunes U, and suggest you check out their detailed specifics on the mpeg-4 format in their guidelines here, if you are getting into Video. Again, I came upon Leigh 's blog via the indispensible Stephen Downes on OLDaily.

Video is a huge subject, and I'll blog a fair bit more about it later. I've have really come to it recently, and have had the pleasure of making all the usual mistakes and learning things the hard way. Alas, If it's worth learning, it's worth learning the hard way.

Norwegians use student laptops for exams

BBC News has the story of a Norwegian trial of using laptops for exams. Online Assessment is nothing new, the technology for it has been part of the web since that start. The hard part is summative, high stakes assessment - Exams. Not only do you need to be able to prevent cheating, you need to be able to have a rock solid, secure, robust process that can withstand Exam board scrutiny. This is tough, and companies like Prometric (who do the Irish driving test exams now) deal with this by running fully proctored exams, as well as using software tools like the one described in the BBC article.

An easier use of online assessment is for formative and low stakes assessment. Little tests along the way help to tell students if they are on track, but they don't contribute enough to the final grade to make them likely to be a point of challenge. Pre-assessment is also a good use of this kind of test - run a little test at the start of the module on things they aught to know before starting your course. It can help them, and you, identify who is in trouble long before the big exam.

Formative and low stakes assessment is a lot easier to manage online. Here in UCC, Blackboard has a tool for running tests like this built right in. Plan on a little time to figure it out and load in the questions, but it isn't difficult to use. Again, if you work in UCC, it's part of my job to help you with this. I'm planning some workshops on it over the summer, please email if you are interested.

Like most Technology Assisted Learning tools, the hard part of Online Assessment is the Assessment bit. Writing good machine-gradable questions is much harder than conventional essay type questions. There is a tendancy there to write questions about recollection of facts, the Pub Quiz /"Who wants to be a millionaire" type question. Sometime this is what you need. It's hard to grasp British History if you don't know when the Battle of Bosworth Field was, but it's harder to write a machine gradable question to find out if they know the what and why of it.

It takes time to write these kinds of questions, but time you will save at the other end in marking. Machine gradable tests, conducted online, can help you understand how larger classes are doing and if they are keeping up without generating hundreds of paper sheets to be marked or scanned. Take some time over the summer to prep some questions, and try it out next term. If you need help (and work in UCC), drop me an eMail r.cosgrave@ucc.ie