The University is changing its approach to module and programme evaluation and has adopted a new tool to deliver it. This approach is being driven by the PVC (Education). The implementation project is being led by Dr Tony Fawcett (Dean of Education), Andrew Rennie (Head of the Curriculum, Learning and Assessment Service), Dr Malcolm Murray (eLearning Manager) and Julie Mulvey (eXplorance Expert). It draws on the findings of the MEQ Task & Finish Group led by Prof Gordon Love. Where decisions are required they have been obtained from the University Education Committee. Data Transfer has been approved by the University Information Governance Unit.
In the past, the University has used a range of tools to deliver module evaluations, most recently Blackboard’s Enterprise Survey Tool. During 2016-17, the University ran a successful year-long pilot of eXplorance Blue with Durham Business School. At the same time, Education Committee ran a Task & Finish Group looking at the module evaluation process. The result was a desire to produce a common set of questions at the level of the University and then within a Department. This will allow aggregation of the data and better reporting. To this end, in May 2017, the University signed a 5 year deal with the Canadian company Explorance Inc. They are a leading provider of course/module evaluation software with clients from across the globe.
The project began in 2017-18 with end of year undergraduate module evaluations. The eXplorance Blue module evaluation solution has been made available to all departments during the Easter term of 2018 and this tool should be used for all evaluations. After completing Undergradute MEQs, the project will then expand to cover Postgraduate MEQs and programme-level evaluations. Staff who need to create evaluations for interim module evaluations or modules that finish early in the academic year) should use Blackboard’s survey tool and are being advised not the Enterprise survey tool. Once eXplorance Blue takes over the MEQ process fully, the Enterprise Survey tool in Blackboard will become read-only.
Enterprise Surveys: Please note that are are not training any new Evaluation Managers for the Enterprise Survey tool and recommend that if you are still using it then you only allow staff who know how to use the tool to have access.
Blackboard Survey tool: This is a tool that sits within each duo modules and allows a survey to be build and run anonymously. You can copy a survey template from one course into others and instructions can be found on the Blackboard Help area within the module.
The New Approach
Standard questions have been defined to provide a more consistent student experience and allow aggregation of the data (so results from modules with low numbers of students can still contribute to a broader, programme-level picture). The question style and responses have been designed to follow those used in the NSS. There was also an acknowledgement that some existing questionnaires were too long, risking a low return rate. Some existing MEQs contained questions that should really be asked earlier in interim evaluations when there is still time to make changes for the current cohort. As such the number of questions has been limited, so that evaluations should be a similar length across the University.
Evaluations will be driven by the data in the student record system. As such we will be evaluating modules by module code, not duo course codes or any other way of aggregating students.
Evaluations are made up of distinct sets of questions. The explanation that follows focuses on the MEQ process. It also applies to programme evaluations – simply replace the word ‘Module’ with ‘Programme’ in the examples below. It uses screen-shots from a bespoke tool (the QuBE) developed by members of the LTT to combine responses from the departments and create the data files used by eXplorance Blue. As such the styling may differ from the final evaluations delivered to students.
The Anatomy of an Evaluation
There are some questions that will be standard to all evaluations delivered across the University. These occur in two places in the evaluation – at the start and at the end.
Here are the opening questions:
These are followed by up to three questions defined by each Department as common to their evaluations. These questions are drawn from a pool of 12 possible questions. Three possible selections are shown here:
Module-specific Questions – Optional
The next block is optional and allows Departments to add up to three Likert-scale questions specific to each module being evaluated. These questions are set by the department – they are free text and so can be tailored to match the learning and teaching experience – e.g. focussing on tutorials, practicals, lab classes, field trips, etc. if they are used in this module. Some sample questions are shown here:
All evaluations include two free-text questions allowing students to comment on positive and negative aspects of the module. These are delivered as separate questions to aid in the generation of reports.
Staff Rating – Optional
Some previous MEQs included a question where students could rate the quality of teaching delivered by staff on the module. In the review of MEQs many versions of this question were found. Some departments reviewed all staff (including tutors, demonstrators, etc.) others only lecturers, yet others no-one. The approach to staff rating was discussed at length at a recent Education Committee, noting the danger of linking this too closely to the probation process. The decision was made to allow an optional single repeating commonly worded staff rating question but that this should be restricted to lecturers only, to keep the size of the MEQ manageable. Departments are able to decide whether or not to include staff rating questions in their MEQs. If they choose to include such questions, they need to provide a list of the staff to rate for each module. An example is shown below:
Equality and Diversity
The questionnaires end with a final institution-wide question asking the student to reflect on their experience on this module. If students answer Yes to the question, they are prompted for more details on a separate page. Note that responses to this sensitive question are not included in the standard module report, they are only visible to Heads of Department and a few others.
Compiling the MEQs
In this first year, we have been developing and rolling out the process as we go. We are very grateful for the timely help provided by staff in each department.
The first step is to determine which modules are running and should be evaluated this year. The Academic Office has generated spreadsheets for each Department listing their modules and providing tools to allow them to select the questions to deploy at both the department and module level. These were then distributed to nominated individuals within each department who had the necessary knowledge and contacts to complete the forms.
Once returned, these spreadsheets are then fed into a bespoke system developed by members of the LTT to generate the feed files for use by eXplorance. Details of the nominated administrators, the teaching staff, the module leaders and students are obtained from relevant University systems. After checking, a series of feed files are generated and supplied securely to eXplorance.
To ensure that the correct modules are evaluated and the correct staff are rated, departments have a chance to preview the configuration before the system goes live. They are able to change the assignment of staff to be rated and the nominated module leaders. This process uses a special tool in eXplorance, known as DIG – the Data Information Gateway. Admin staff have been sent emails providing secure access to this system.
Delivering the MEQs
Students eligible for MEQs will receive an email on the opening day of the evaluation period informing them that they have MEQs to complete and containing a link that allows them to complete them. They will also see the same list of MEQs available to them on a special MEQ tab in duo (handy if they later misplace the email):
Once the evaluation period is over, reports will be automatically generated for each evaluation. The exact nature of the reports is still being considered but we expect the list will be something like this:
- Individual Staff Member Report – showing the results from the standard likert scale questions for the module, (plus the responses to this person’s staff rating question, if it is deployed).
- Module-Wide Report – showing the results of all MEQ questions (except the equality and diversity question) this is available to the nominated Module Leader and Head of Department.
- Equality & Diversity Report – showing the results of positive responses to this question and detail of the free text responses. This would only be made available to a very restricted set of individuals in the first instance – Heads of Departments and staff involved in supporting Equality and Diversity across the University.
Other aggregate reports are being developed (e.g. results across a department and programme). It has also been agreed to release reports for students, listing the results of the institutional and departmental likert questions (but not staff ratings, the equality and diversity question or any of the free text responses).
FAQs re Module Evaluations
Below is a list of questions that may be asked about the new Module Evaluation process
We are running Undergraduate end of module evaluations from Monday, 23rd April (2.00 pm) until midnight on Sunday, 6th May. Reminders will be sent periodically throughout this period to those students still to complete evaluations.
The University requires a minimum 50% return rate for each end of of module evaluation, which is in line with return rates for the NSS (National Student Survey).
Students will be sent notification emails (sent from firstname.lastname@example.org) which will include a link to a new system which lists the evaluations you need to complete.
The new evaluations have been made shorter and more consistent so should take less time to complete. Evaluations can be completed on a pc or laptop, mobile or tablet device of your choice.
The University have decided on a focussed two week window, just before the exam period, in which we encourage students to take the time to give us their feedback. Reminders will be sent to those students with surveys still to complete.
Students can also locate the evaluations they need to complete by clicking on the dedicated MEQ (Module Evaluation Questionnaire) tab in duo. Students will also be able to see the percentage return rate. Please help to increase the return rates to at least 50% by giving the University your feedback.
The surveys have been developed in conjunction with your Department/School and include questions on an institutional level, departmental level as well as a module level. In some Departments/Schools a decision was made to include a core staff rating question and in others to exclude it for this year. So this year you may or may not be rated.
If you login to duo you can locate the MEQ tab – this will allow you to view the evaluations list for the modules that you teach on or are module leader on. Please view the Return Rate monitor for each evaluation. Your involvement is vital in helping the evaluations to reach the required return rate. When you meet with your students during the next few weeks please encourage them to take the time to complete their evaluations. You may want to allow 10 minutes at the beginning or end of your session to allow the students to access their evalutions on a mobile or tablet – they can search for an email from email@example.com or locate the MEQ tab in duo. You may want to explain the importance of giving feedback on the modules they have taken which will help the direction of teaching in the future.
If you have a question regarding the new Evaluation process or you have any problems completing surveys please email firstname.lastname@example.org and put “Explorance Blue” in the email subject field.
The different types of reports that will be available after the evaluation period and these will be located on the MEQ tab.
We will be developing not only reports for Teaching Staff, Module Leaders, Heads of Departments, Aggregate raw data reports for Administrators but also a report for Students.
Please note that we do not have a date for the release of the reports at this moment in time as we are currently developing them.
The University (Curriculum, Learning & Assessment Service) has taken that for MEQs that receive less than 5 responses, that reports will not be run, in order to preserve the anonymity of the students providing responses. So if you see the status Not Met – this means that it did not reach the threshold.
As a member of staff looking for a staff rating response, please check that your Department/School asked for that question to be included.
If they did, ask them to confirm that your details were passed along.
Please note that if your involvement was as a tutor or seminar leader then you will not have been included in the process.
To keep the evaluations short we limited the staff rating question only to members of staff who were Lecturers and that the students will have been involved with.
If you have any questions about eXplorance Blue please email email@example.com and put “MEQ Advice” in the subject line.
Liverpool John Moores University amplifies the power of module evaluation data with Blue – READ
Alternative tools you might want to consider for interim evaluations include:
- Blackboard Survey Tool – the course tool that sits in duo
- Bristol Online Survey – the University has a site-license for staff and research students. Accounts are available by contacting the IT Service Desk. The data sits securely in the UK.
Please note that we do not advocate the use of other cloud based products that have not been cleared by the University – so please check with your Head of Department or the University Information Governance service before using anything not listed above.
REVIEW DATE: April 2018
PRODUCT: Blue 7