Academics and students in conversation about multimodal assessment
Mira Vogel, Senior Teaching Fellow, King's College London
UCL Education Conference
17th April 2018
3.30pm Rm 802
"Students learn to produce outputs - assessments directed at an audience" is one of the dimensions of the UCL Connected Curriculum. It implies a diversification of student work into media and genres beyond the essay, report and test. How does this play out for students required to produce digital outputs such as blogs, videos websites, and podcasts? How can assessments be designed to encourage students be ambitious but not leave them floundering? How can peers help each other achieve more than can be achieved alone? How do assessors - whether academics or fellow students - go about making judgments about diverse interpretations of the same assessment brief, exhibiting diverse skills? How do assessors manage the changes?
Drawing on findings and recorded first person accounts from a 2016-17 Connected Curriculum Fellowship project (https://wiki.ucl.ac.uk/x/LUq_Aw) and a review of the relevant literature, this presentation will consider these questions.
The Prezi version
The accessible text version
I'm going to talk about making and assessing multimodal work.
By multimodal, I mean work produced in several different media - I'll concentrate on digital media, including video, audio, and graphics as well as alphabetic texts - and deploying different genres including blogging, podcasting, explainer videos.
The origins of digital multimodal assessment are often traced back to the New London Group of university educationalists, who met in 1994
"As soon as our sights are set on the objective of creating the learning condition for full social participation, the issue of differences becomes critically important."
Dilly Fung set out the elements of a connected curriculum in 2017
"... assessed activities to mirror the kinds
of communications, including public engagement activities, that are undertaken by researchers and enquiring professionals in many fields." Fung, 2017
This was timely because in 2016 Fraser Nelson, editor of The Spectator, had released this advert for interns.
Since then 2017 Obasi Shaw has submitted Harvard's first rap thesis combining elements of Middle English poetry with issues of racial identity in the US. And the Times Higher continues to meditate on whether a film can count as research.
In 2016 I successfully proposed a Connected Curriculum Project which I completed in summer 2017. This entailed recruited 11 pairs, sometimes threes of staff assessors and students, to discuss the task, the process of producing the work, and the process of assessing it.
For this presentation I'm going to concentrate on two aspects:
- How are students be supported to produce good academic work in unfamiliar, informal settings?
- How do assessors judge the diverse interpretations of a task?
This is based on transcribing the interviews and analysing them inductively in Nvivo.
I also thought that, as somebody, enthusing to colleagues about multimodal assessment, I had better as they say 'eat my own dogfood', so I produced my report as 7 thematic videos which I'll excerpt here and link to at the end.
A major theme which came out of the analysis is the power of peers.
Peers take an interest in each other and spread ideas
Power of peers https://youtu.be/Uy7wzKf6MtE?t=41s
Kerstin: … I thought there was some real, very good graphics coming out of that. And one student doing that in one week, very effectively. You could then see two weeks later it had this cascade of people thinking “Oh this is really!..” So it’s in that sense that some of the ideas of how to convey a message - not so the content, because obviously that was very specific and individual - but more the kind of “Oh, this is a nice idea”. Which I think that’s what learning is about. So from that perspective I was very happy with that.
Oreoluwa: I did learn how to, learned by watching - even from the individual videos, watching other people’s videos, you were like “Ohhh, I could do that!”, “I should have done that”, because I feel like mine was a lot of text, because I didn’t know how to make funky graphics. But when you see other people’s videos you say “OK, I can do that, I can do that”. So I did learn. [CUT IN STILL OF FUNKY GRAPHIC FROM THE VIDEO]
Peers sustain each other in ambitious interpretations of the task
https://youtu.be/Uy7wzKf6MtE?t=2m35s to 4m37s
Samir: It would it would be really difficult to make an engaging interesting video on your own, but if you have a group you can not only can you bounce ideas of multiple people, you make it together and there's everyone's contributions. Like all the random props we used were just sourced from whatever people could find, and the ideas all - everyone had some little bit they added to it, and it wouldn't really be easy if we'd done it by ourselves [CUT IN STILLS OF THE GROUP IN THE VIDEO INCLUDING THE PROPS]
Arantza: So I remember we got really excited and we had a very long conversation, about an hour or so, about how we would imagine a black room, kind of dim lights walking you through into the first object, without being able to see the rest of the room, and then that object lighting up and being able to read the description on it, and we just found that really interesting. And then when we went to put that into virtual space, we found that it was a bit difficult, but it could be done through putting up a video and then a description of what we as the creators imagined the exhibition to look like. [CUT IN STILLS FROM THE EXHIBITION SPACE AND VIDEO]
Katie: I just hate group projects, I do. But in the end it went really well, yeah. We all wanted to do a really good job and that was the main thing. So we focused on that.
Ruxandra: I feel like we just worked hard on this. It got to the point where it was like midnight and we had another idea and we were like “That’s actually so much better than everything we came up with”. And we worked with that idea.
Informal doesn't equal unacademic
Students need support to avoid dichotomising academic work on the one hand, and engaging creative work on the other.
Julien: I remember after those practice presentations you’d always be saying “You’re waffling” or “You haven’t actually explained to the lay viewer exactly what you mean” and that’s because when you’re writing an academic essay I think you kind of assume … You assume certain knowledge – you don’t really think about it.
Jacky: Yes. Yes! You do, and you also assume your audience understands the words that you are using or the people that you’re quoting. Whereas if you’re talking to an audience that we have said is general public, but intelligent, you have to make sure that the line that you’re following is one that they can follow too.
Students need support to understand why they are asked to produce the new forms of work
Katarina: I imagined something like those TV series where they sometimes change the facts… But here after some time I found that the more research we do, the better the work gets and it actually becomes an academic piece of work.
Julien: I was quite cynical, I think, when I heard the whole idea. I’m not really sure I made the connection between… I know we were told, “This is public history”, therefore you kind of make it more accessible. But I kind of dismissed it. I think the first two weeks, maybe, is just… an attempt to try to bring technology to history, which I didn’t feel was something that was needed at all … Just the emphasis on… We were told “You’re going to use a website, you're going to make a video”. I mean, having done it now, I see what the actual point of it actually was. Whereas at the beginning I was just like “this is just part of this whole craze of integrating tech into [the curriculum]” – but it wasn’t that at all.
The abstract asked how assessors manage the diversity. The first thing to say is that the assessors find marking exhilarating because of the diversity
https://youtu.be/RXEC_8Eki6A?t=2m to 3m28
(3 clips - or 4 with Adam)
Hannah: Reading through all of the projects, and looking at all the projects, it’s rewarding to see where people have taken it. These kinds of projects that you’re doing are quite innovative and I think that’s really exciting. Learning ourselves from the students, about some of the possibilities that they’ve managed to create by bringing these methods together in relation to their own field site.
Jacky: And we had some amazing things where we were “My goodness how did they do that?!” But at the beginning of the course, “We’re historians, we don’t know how to do this”. But the work produced at the end, it’s just incredible. I’m always amazed.
Thomas: I think the diversity is a benefit in many ways. You know, it’s brilliant for us to see, and it’s much more interesting than reading 40 or 35 essays essentially on the same or very similar things.
Adam: all the blogs that have been made have been really impressive...
Form or content?
But then we come to consider what is recognised, what gets credit - how much for the substance and how much for the form it takes?
Here are Antony and Laura from Greek and Latin, setting out the dilemma
Antony: I think if you give choice there, if somebody presents a PowerPoint or someone presents a Prezi, the tutor might get carried away and assign more marks to the person who has done for example, the Prezi, without necessarily the student deserving the marks, but the Prezi deserving the marks. You might get carried away and give more marks to the actual mode rather than to the content of what the students have done.
Laura: However, you could argue that the aim - obviously to share this knowledge with the audience and the students, if the goal was for them to have a wider understanding of this particular topic, and it worked, then surely that can be taken into consideration in a way. With the end goal insight.
This contrasts with the Academic Composition community in the US higher education system.
Chanon Adsanatham wrote a stand-out paper titled 'Integrating Assessment and Instruction: Using Student-Generated Grading Criteria to Evaluate Multimodal Digital Projects'. His students spent weeks watching examples of multimodal academic communications and assessing them with these questions:
"Who is the audience? What is the context of the argument? How are images arranged? In what order do they appear? Is there any logic to them? What rhetorical appeals are utilized? What sounds are heard first, next, and afterward, and why are they arranged in that way?"
Then students negotiated their own set of criteria together before producing their own multimodal assignments.
I don't want to argue that academics in the disciplines should be doing all of this, since Academic Composition assessors are prioritising the rhetorical features and relegating the subject knowledge.
However, without this willingness to recognise these aspects, the assessment criteria are liable to remain very similar as for print media, there's trepidation about rewarding students' digital capabilities, and there's there's a reversion to alphabetic writing as favoured vehicle for meaning - here in the form of the 2,500 reflective essay.
- Negotiating changes to uni assessment criteria is often hard.
- Primarily, university essays have been assessed as vehicles for knowledge rather than exercises in communication.
- In contrast, the academic composition community centres on rhetorical aspects - high level criteria which in turn depend on students' concepts of audience and competence with tech.
- Conceptualising audience is one way to keep subject knowledge central while making space to recognise rhetorical aspects.
- This adds complexity, but groups sustain each other with skills, vision and companionship.
- Negotiated criteria, examples and guided marking help students avoid dichotomising academic writing and multimodal authoring.
- Assessing groups - peer assessment of contribution (including UCL IPAC https://wiki.ucl.ac.uk/x/igK_Aw).