The Highs and Lows of the Educational Innovator: of cobbles, haircuts, and flipped assessment (Part One)

Chris Froome ‘Tour De France 2013’ image credit: https://www.flickr.com/photos/sumofmarc/9347911050
Let me share a tale of blood and toil on the frontline of educational innovation.

I teach a 100+ student second and masters year Economics unit at Monash University that I built from the ground up.
After seven years of hard work, carefully tending to my unit’s content, assessments, delivery method and engagement mechanisms the fruit had started to come —
Purple letters of commendation from the VC; winsome comments from chance-meetings with long-past students on how the unit changed their life and career direction; requests from various faculties to bring the unit into their degree structures (with mixed success, university bureaucracy being what it is) .. you may have supped of such things.
Metaphorically, in lock-step with Chris Froome’s many years’ toil and sacrifice, by 2013, my unit had finally won the equivalent of the Tour de France. And like Froome, so said the pundits (and I dared believe myself), now all that remained, was to enjoy back-to-back educational victories that would come my way.
But one cold morning in July 2014, the honey stopped.
It was Unit Evaluation results day, and as much as we say it doesn’t matter, I’d been given a razor-line haircut across the board.
As it turned out, Froome’s July wasn’t going any better: on a non-discript piece of tarmac near the famous Arenberg cobbles in Northern France,Chris Froome crashed out of the 2014 Tour, bloodied and broken. .. The parallel was hard to miss.
What happened?
It wasn’t incessant rain, impossible road furniture, or even fabled cobbles which brought me down.
My sin was more simple: innovation. I tried something new.
I’m a repeat offender actually, I like innovation (I even formally study it), and I’ve been innovating every year I’ve been on the faculty.
But this time, I went for something really big. Radical even. I decided to turn over the marking of the in-semester essay to the students.
That’s right: I was going to embed Peer-Review in my unit.
My reasoning was pretty simple. Academic scholarship requires performing critical review of peers’ work. In fact, life requires it — any professional job that my students were going to take would require them to provide constructive, thoughtful and critical feedback to their peers.
But we don’t teach critical peer review.
I know this because I asked my students. Of the cohort’s response to my anonymous survey, exactly zero undergrad students had conducted a peer-review task previously. Of the post-grads, peer review was new to four in five.
So I decided to ‘flip’ the assessment and put the students in charge.
Practically, this meant spending a lot of hours (I estimate about twice what it would have taken me to mark every essay submitted by hand ..) to research, trial, implement, test, refine, deploy, communicate (and re-communicate), and run two peer-assessment modules in Moodle (one for each within-semester assignment).
I took the research and suggestions on Peer Review very seriously: I used Moodle tricks to get double-blind peer-review; I defined and tested abinary marking rubric for each assignment; I provided multiple pre-assessed examples for each of the ‘training’ phases (and made it arequirement for every student to train on a minimum of these); I posted lengthy FAQs to all kinds of imagined questions to communicate thewhy and how of the process; I ensured every student received at least three peer-assessments on their work (to minimise the impact of ‘rogue’ assessments); I provided a moderation and appeal process .. you get the picture. I thought I was pretty diligent.
In the end, around a third of the students thought I’d done something terrific by bringing in peer review, about a third were lukewarm about it, and the remainder had struck me off their Christmas Card list. And they told me.
The key issue seemed to be the ‘peer-ness’ of the assessment. A good number of my students railed against the notion that their ‘academic assessment’ would come down to a weighted (moderated, appealed) combination of their peer’s assessments.
As an economist, trained to look past stated preferences, and instead observe what people do,  part of me didn’t believe them: of the 100 or so students, only seven opted to have their peer assessments reviewed by me in the appeal process. Of these, only one mark was adjusted up because of my review. I took from this that 93% were satisfied with their mark, and that of the remainder, in over 85% of cases my assessment agreed with the average of their peers (just like the literature said).
As an educator, I was worried by the feedback. It seemed that even though people weren’t coming forward for a re-mark, the angry third had two big issues: the perceived poor quality of the comments they received from peers (even if they agreed with the marks); and/or the large variation in the marks their peers gave (even if the average seemed reasonable).
Partly, this is to be expected. Students aren’t expert markers. They are bound to make over-corrections, find it emotionally difficult to mark down a peer (even an anonymous one), apply the criteria inconsistently as they develop their own understanding of what the criteria is asking … and so on.
Partly, this is exactly what I wanted the students to experience — this is live learning. This is why I wanted it to be summative. Some students surely felt the responsibility keenly, and in the vast majority of cases I was tremendously impressed by the maturity and professionalism of the cohort.
But still … my radical change was radically risky. Consequently, I spent that 2014 semester in a yo-yoing state of dread and exhilaration.
What to do? Fold, and so push Peer Review to the ‘not worth the trouble’ pile and go back to Faculty-marked essays? Replace, and so exchange Peer Review for a different system altogether?, or Press On, and steel myself for another semester of pain!?
… next time, I’ll explore my response, and draw some lessons for educational innovation. Click here for Part Two…

 

Creative Commons License
The Highs and Lows of the Educational Innovator: of cobbles, haircuts, and flipped assessment (Part One) by Dr Simon Angus is licensed under a Creative Commons Attribution-NoDerivatives 4.0 International License.