M37 - Moodle Workshop for peer assessment

Keywords: workshop, peer assessment, peer feedback, self-assessment, self assessment, peer marking

What is it?

Workshop is one way Moodle staff can set up student peer marking in Moodle. The activity can accommodate different kinds of submitted work and can happen before, alongside, or independently of the standard tutor assessment. Workshop lets staff set up forms, rubrics or marking guides to support students in making judgements. Student assessments can receive feedback only, or feedback and numeric grades. Tutors can optionally assess the student submissions and assessments, and weigh their own judgements in peer relations.  Peer submissions and assessments can be kept anonymous if needed. Workshop also allows self-assessment.

The basic stages of a Workshop activity, set out in more detail below, are:

1. Staff set up a Workshop assignment to which students will submit their work. This includes authoring instructions for submission and marking, setting how many pieces of work each student marks, by when, how allocated, whether anonymous, whether feedback only or numerically graded.
2. Students submit their work.
3. Students mark each others' work.
4. Tutors review the submissions and assessments, and optionally give their own marks for these.
5. Students receive the feedback and/or marks given by their peers and (if used) a mark from their Tutor for the assessments they have given.
6. A Tutor closes the activity, at which point the marks and feedback are revealed.

Why use it?

Peer marking requires students to make judgements about the work of their peers. Peer markers may be expected to give feedback only (which we'll refer to here as 'peer feedback'), or feedback along with a numeric mark (which we'll refer to as 'peer assessment').

A well-conceived, well-enacted peer assessment activity can advance:

• Students' ability to understand and work with assessment criteria.
• Students' ability in the authentic academic practice of peer review.
• Insights, through articulating judgements and producing constructive feedback, about how students can go about critiquing and improving their own work.
• The possibility of feedback that is quicker, more individualised, and more plentiful than tutors are able to provide. 
• The possibility of feedback on students' draft work, with sufficient time for amendments before its deadline.
• Avoiding 'learned dependence' (Yorke, 2003) - students' over-reliance on tutor opinions, and over-humility to the importance of their own understandings.
• Triangulation - the original submission, peer reviews and tutor assessment (not to mention self assessment where used) can be compared, giving students new perspectives on their submission, the criteria, and the reviews they have written.
• Relatedly, insights into subjectivity and governance in the assessment process.
• Also relatedly, a departure from monologic, transmissive feedback as students weigh up the differences in the reviews. This in turn promises a desirable change in the way feedback is received from simple certainties to more sophisticated, evaluative thinking (Schommer, 1990).
• An occasion for dialogue with tutors and peers about assessment.

Keith Topping (2009) suggests explaining to students,

'...that peer assessment involves students directly in learning, and should promote a sense of ownership, personal responsibility, and motivation. Teachers can also point out that peer assessment can increase variety and interest, activity and interactivity, identification and bonding, self-confidence, and empathy with others.'

Before I start...

• First time use of any new activity needs time and careful planning - especially one this coordinated and interpersonal. As you and your students repeat the process, it becomes far quicker and easier to run than the first time.
• Peer marking activities stand or fall on the strength of the explanations and instructions you give to students. These relate to the assessment criteria and the underlying theory of peer marking, as well as to the technology.
• Though the Moodle Workshop activity presents staff with a clear dashboard list (called the Planner) of who has and hasn't completed the two stages, it doesn't currently let you generate the same kind of participation reports as other Moodle activities. So if you have a very large cohort, we suggest using groups to make the list more manageable.

How do I set one up?

Creating the Activity

1. Switch the toggle to turn 'edit mode' on.  
2. Hover your mouse of the area of the course where you would like the Workshop activity to appear, and click the '+' Plus symbol (or go to the bottom of the section and select 'Add an activity or resource'),
3. Select 'Workshop'.
4. Complete the settings as detailed below. Keep in mind that what you put in the text fields of the settings has a crucial orientation role, displaying at certain times in certain places. We give guidance below.
5. Workshop name - this will display as the link to the activity, so make it concise and meaningful.

Description

 This displays for students on the front page of the activity just under the Planner (dashboard). Give a brief motivating introduction to the activity. (Instructions for each phase come later).

Grading settings

Grading strategy determines the assessment form used and the method of grading submissions.

There are 4 alternative grading strategies:

• Accumulative grading - each assessment criterion has its own numeric grade along with optional weighting and optional comments; a final grade is calculated on the basis of the separate grades and their respective weightings.
• Comments - no numeric grade but feedback only, as comments either in a single field or as responses to a series of guiding questions.
• Number of errors - markers decide whether the work they are marking has passed or failed each criterion (e.g. has original ideas; answers its question).
• Rubric - generates a numeric grade based on the level of achievement markers choose for each criterion. You'll be able to define your criteria and, for each criterion, as many levels as you need. The Rubric will display with a free text field for Overall Feedback. Do note that Rubric will generate a numeric grade and cannot be set to feedback only. In other words you will need to assign a single numeric grade to each level, and those marks must be unique. Note too that it is not currently possible to import or reuse rubrics created elsewhere.

Grade for submission and Grade for assessment allow you to set a maximum grade which can be obtained on a piece of work.

The Submission grade to pass and Assessment grade to pass boxes allow you to set a minimum grade required for students to pass the assessment. 

Submission settings

Instructions for submission display to students above the place where they submit their work. It is a good place to explain the settings and any conditions you have set up.

Example submission instructions:

Submit your work below [link to guidance] Please note that there are no late submissions for this activity - you need to submit work before the deadline in order to be allocated others' work to mark, and in order to receive marks and feedback yourself.

Maximum number of submission attachments - if set to none then students will need to paste their submission directly into a Moodle text field. Since Workshop does not have autosave, you might suggest they draft elsewhere and paste into Moodle. Alternatively you can allow files, in which case let students know how many you expect and in which format.

Submission attachment allowed file types - leaving this field blank allows students to upload all file types. Clicking 'Choose' allows you to create a list of allowed file types for student submission.

Late submissions - if late submissions are allowed there is no way to automatically allocate peer markers to them. You'll need to to manually allocate markers to these late submissions, or mark them yourself.

Assessment settings

Instructions for assessment - this displays to students during the assessment phase and appears just under the Planner and above their allocation. Like the instructions for submission they are important for orienting and motivating students.

Example assessment instructions:

Immediately after the submission deadline you will be asked to complete a practice assessment. Once you've done this you'll be able to start giving feedback on fellow students' work you have been assigned, until the assessment deadline. 

The submissions have been assigned to you randomly and automatically.

Use self-assessment - this allows students to be allocated their own work to assess.

Feedback 

Overall feedback mode - provides a field for students to type summary comments which they have not made elsewhere.

Maximum number of overall feedback attachments - this may be none. Attachments come in handy if you are, asking peer markers to comment directly onto the work. There's no need to ask peer markers to upload a completed marksheet here, since Moodle Workshop provides you with ways to make these (see below).

Conclusion - this displays to students after the activity has been closed i.e. when they can come and collect their feedback and/or marks. It appears just under the Planner and above their feedback.

Example submissions

Example submissions are a practice assessment opportunity for students.

Use examples - this will allow you to give students a piece of work so they can practice peer marking without their feedback reaching a genuine peer. You will also provide an exemplary assessment for them as a reference.

Modes of examples assessment - choose when students encounter the example, and whether they have to mark it.

Availability

Here you set dates for the Submission and Assessment phases.

Switch to the next phase after the submissions deadline setting - this begins the Assessment phase automatically at the Submission deadline (the alternative is to manually switch phases when you're ready). If you are going to switch automatically you will almost certainly need to set up Scheduled Allocation - see below.

Common module settings 

These are the same as other Moodle activities.

You will be able to edit many of these settings retrospectively, but be aware that many of them display in the students' Planner view, who may plan around them - so once the Submission Phase is underway it's best to leave the common module settings alone unless you have a chance to negotiate changes with students.

When you are ready click 'Save and Display'; the Setup Phase page displays.

Setup phase

During this phase, only Moodle editors are active - students aren't involved yet.

The example Planner shown below is telling us that before we switch to the Submission Phase, we first need to edit the Assessment Form and prepare Example Submissions. Under that, the Description that was added in the Workshop Settings will be displayed.

Editing the Assessment Form

To set up the assessment, in the Setup phase panel on the left, click the link to Edit Assessment Form; what displays here depends on the grading strategy you chose in the Grade settings component of the Workshop settings. (You can also change this during the Setup Phase).

If you chose Accumulative Grading within the grade settings of the Workshop, you'll be asked to define one or more aspects. For each criterion aspect you'll specify the maximum points available or which scale to use. You can have different points or scales for each aspect. You'll also be asked to give each aspect a weighting in relation to the others. Peer markers will see the definition of the aspect, a place to enter the grade or scale, and a field for their comments.

If you chose Comments within the grade settings of the Workshop, you'll be asked to define one or more aspects. These are the questions or feedback points which will display to peer markers, with a blank text field under each. This is a feedback-only marking strategy so there are no numeric grades or weightings to set.

If you chose Number of Errors within the Grade settings of the Workshop, you'll be asked to define one or more assertions along with words for peer markers to use to indicate whether these have been met or not met. You'll also be asked to give each assertion a weighting in relation to the others. Save And Continue Editing to activate the Grade Mapping Table, which turns the weightings you have chosen into a number of possible errors and relates these to the overall number of points available for the Workshop activity - so what you then need to do is make sure that the values in each menu is the same as their adjacent values.

If you chose Rubric within the grade settings of the Workshop, you'll be asked to define your criteria and levels of achievement. You need to assign a unique grade to each level. Extra levels will appear 'on demand' - first complete those that display by default, then click Save And Continue Editing. When the page reloads it will display extra blank levels. You will be able to choose whether the rubric should display to students as a (vertical) list or (horizontal) grid (list is best where there are many levels).

When you are ready you will need to manually enable the Submission Phase - during this phase students submit their work. Only after the Submission Phase deadline do students begin to peer mark.

Adding Example Submissions

Still in the Setup Phase - if, in the initial Workshop Settings, you enabled the example submission, the Planner will now prompt you to add one by clicking the button.

You are then prompted to paste or type an example submission directly into Moodle, which you then 'mark' in an exemplary way according to your chosen grading strategy. This becomes a reference for students. If you want to show students a range of work, you can add more than one example submission (If you do add more than one example submission, it is probably not a good idea to force the student to assess them all before being able to proceed).

When a student assesses the example, their assessment does not accrue marks. They are shown the example submission and the Assessment Form we set up earlier, and after completing the form and saving the assessment, they are presented with the tutor's reference assessment for comparison.

Students have the opportunity to re-assess the example.

By default, students are anonymous, both as authors and markers

Students tend to ask for anonymity, both for their submissions and assessments. Our evaluations of Workshop have indicated that without anonymity (and depending on the experience and trust among students) 'everyone is holding back' on passing judgement on their peers' work. This is to be expected - at least while they are inexperienced at peer marking. Anonymity is something to negotiate with students and be very clear about.

In the event that a workshop doesn't require it, here is how to remove anonymity for both Author and Marker (which Moodle Workshop refers to as 'Reviewer').

1. In the Workshop, from the More menu, select 'Permissions'; the Permissions page for that particular Workshop activity will display.
2. Filter by the word: view.
3. From the shortened list, find the permission 'View Author Names' and after clicking the '+' icon select 'Student' from the list of available roles, repeat this last action for the 'View reviewer name' permission.
4. Finally return to the Workshop front page by clicking the Back To Workshop... link at the bottom.

Submission phase

It often helps to send a News Forum message reminding students that the submission phase is underway, its duration, and where to find it.

To monitor submissions, you can refer to the Planner to monitor how many students have submitted work so far, and below that you can filter by Group (if you've set them up).

Allocate submissions (before the Assessment Phase)

Allocation of submissions to peer markers could be done as student submissions arrive, but it is probably easiest to wait until after the deadline - and easier still to set Moodle Workshop up to do this automatically. 

Scheduled allocation (automatic after the Submission Phase ends)

You may not want to do this if you are expecting students to submit late. However, it does offer the benefits of moving things smartly along and allowing students to peer mark while their own work is still fresh in their minds.

The most automated this gets is as follows - in the Workshop settings, in the Availability section:

• 1. Ensure you have set a Submissions deadline date.
  2. Ensure you have checked the box titled Switch to the next phase after the submissions deadline.
  3. Ensure you have set an Open for assessment from date.
  4. Since these dates display to students, also ensure that you have set a Deadline for assessment.
  5. Click 'Save and Display.' The Planner will display.

There's one more thing to do for these settings to be enacted.

1. From the Workshop menu at the top of the screen, select Submissions allocation and configure as follows:
2. Select the Scheduled Allocation option from the drop down list to see the available Scheduled Allocation settings.
3. Select 'Expand all'.
4. Tick the tick box Automatically allocate submissions at the end of the submission phase.
5. Decide on the Number of reviews (assessments) which should be made (per peer marker or per submission, depending on your preference). How many is enough? You need to predict whether all or just some students will do the peer marking. You want to avoid overloading students while ensuring that everybody gets some peer feedback. Our evaluations suggest students place a higher value on the feedback they receive than the feedback they give, even though the latter is considered more educationally potent.
6. Decide whether students should be able to peer mark if they themselves have not made a submission.

7. Decide whether students should self-assess.

   Checking the Add Self-Assessments box will assign peer markers their own Submission in addition so keep students' workload in mind when you set the number of Reviews.
8. Save Changes, and then click on Submissions allocation again to check things are as you intended. There should be some ticks and confirmations. Set up as illustrated below and you will not have to manually progress the activity from Submission Phase to Assessment Phase..

Manual allocation

You will see a list of the participants (students on the course), and to the left a column to assign them a reviewer, and a right column to assign them a reviewee. A participant must submit something before they can be assigned a reviewer, however you can assign students to review someone else's work even if they have not submitted themselves. 

Random allocation

The other way you can allocate reviewers and reviewees is by random allocation. If you have groups set up on the Workshop activity then you can set the allocations to distribute among group members. You set how many students you want to review each piece of work. You can choose to override any previous allocations, for example if you have some extra submissions you may wish to re-do the random allocations. There is also the option to allow those who have not submitted to be reviewers, if you do not tick this then anyone who has not submitted will not be allocated a reviewee. Once you have selected the relevant settings click 'Save changes'.
When the random allocation has completed you will get a confirmation screen, which will also note any problems, such as not being able to allocate to individuals who have not submitted.

Assessment phase

Once submissions are completed and reviewers have been allocated it is time to move into the Assessment phase. This can be done manually, but if you set it up in the initial settings then this will happen automatically at the set date. From a tutor or course administrator perspective there is not much more to do at the Assessment phase, once you have activated it. This is the students' time to review their peers work and whilst doing so also reflect on their own pieces. 
 
As the grades or comments are given by peers to one another you will start to see these appearing in the table. This will allow you to chase up anyone who has not peer marked before the assessment phase closes. The names highlighted in red are of those still to give feedback, the ones in black have completed, and in the case where grades are given you can see what they have given. 

Grading and evaluation phase

This phase is all about the tutors or course administrators, as it is your chance to review the comments and grades given by students and make any amendments where they have been either too lenient or too harsh. It basically enables you to mitigate any negativity and ensure all grading has been done fairly. You can also allocate marks for the reviewers based on the quality of their reviews if you wish to. 

Closed phase

Once you turn on the Closed phase then the students can see any grades or comments they have been left by the other students. This is also the end of the process.

Caution

• Setting the Workshop activity to automatically switch phase from submission to assessment and automatically make allocations, is one of the most labour saving aspects of Moodle Workshop. However, it also means that if there are submissions after the allocations have been made, a tutor needs to make manual allocations of that work to assessors. Tutors need to work out a process to handle this, especially if the activity is compulsory and credit bearing, because reviewers shouldn't be surprised by extra allocations late in the day. One way is to build in a few days' delay before switching from submission to assessment and running the allocation. Another complementary way is to set the activity to allow assessment by students who have not themselves made submissions - that way the student will be allocated work to assess, and their own work can be marked by the tutor.
• To recognise and incentivise the assessments, if using numeric marks (rather than feedback only), make available a proportion of credit for the assessment as well as - or even instead of - the submission.
• Include instructions in the settings of the Workshop. Pay attention to how the instructions display for students and adjust them accordingly.
• Tutors make diary entries to monitor the activity and (notwithstanding the Upcoming Events Block) to send students reminders a few days before the submission and assessment deadlines, and after the activity is closed. If students are new to the activity, do include instructions about where they can find their allocated work and their feedback. Use the Announcements or News Forum for this (unfortunately since Moodle Workshop is multi-phased, Moodle Course Participation report doesn't help us here).
• Anonymity - do go into the Workshop Permissions and change the Permission for View Author Names so that marking is blind; this reduces the politics and social discomfort of passing judgement, and makes peer markers more comfortable about giving constructive criticism rather than platitudes.