eXplorance Blue Course Evaluations

The University is changing its approach to module and programme evaluation and has adopted a new tool to deliver it. This approach is being driven by the PVC (Education). The implementation project is being led by Dr Tony Fawcett (Dean of Education), Andrew Rennie (Head of the Curriculum, Learning and Assessment Service), Dr Malcolm Murray (eLearning Manager) and Julie Mulvey (eXplorance Expert). It draws on the findings of the MEQ Task & Finish Group led by Prof Gordon Love. Where decisions are required they have been obtained from the University Education Committee. Data Transfer has been approved by the University Information Governance Unit.

In the past, the University has used a range of tools to deliver module evaluations, most recently Blackboard’s Enterprise Survey Tool. During 2016-17, the University ran a successful year-long pilot of eXplorance Blue with Durham Business School. At the same time, Education Committee ran a Task & Finish Group looking at the module evaluation process. The result was a desire to produce a common set of questions at the level of the University and then within a Department. This will allow aggregation of the data and better reporting. To this end, in May 2017, the University signed a 5 year deal with the Canadian company Explorance Inc. They are a leading provider of course/module evaluation software with clients from across the globe.

Implementation Timetable

The project began in 2017-18 with end of year undergraduate module evaluations. The  eXplorance Blue module evaluation solution has been made available to  all departments during the Easter term of 2018 and this tool should be used for all evaluations. After completing Undergradute MEQs, the project will then expand to cover Postgraduate MEQs and programme-level evaluations.  Staff who need to create evaluations for interim module evaluations or modules that finish early in the academic year) should use Blackboard’s survey tool and are being advised not the Enterprise survey tool.  Once eXplorance Blue takes over the MEQ process fully, the Enterprise Survey tool in Blackboard will become read-only.

Enterprise Surveys:  Please note that are are not training any new Evaluation Managers for the Enterprise Survey tool and recommend that if you are still using it then you only allow staff who know how to use the tool to have access.

Blackboard Survey tool:  This is a tool that sits within each duo modules and allows a survey to be build and run anonymously.  You can copy a survey template from one course into others and instructions can be found on the Blackboard Help area within the module.

The New Approach

Structured Questions

Standard questions have been defined to provide a more consistent student experience and allow aggregation of the data (so results from modules with low numbers of students can still contribute to a broader, programme-level picture). The question style and responses have been designed to follow those used in the NSS. There was also an acknowledgement that some existing questionnaires were too long, risking a low return rate. Some existing MEQs contained questions that should really be asked earlier in interim evaluations when there is still time to make changes for the current cohort. As such the number of questions has been limited, so that evaluations should be a similar length across the University.

Evaluations will be driven by the data in the student record system. As such we will be evaluating modules by module code, not duo course codes or any other way of aggregating students.

Evaluations are made up of distinct sets of questions. The explanation that follows focuses on the MEQ process. It also applies to programme evaluations – simply replace the word ‘Module’ with ‘Programme’ in the examples below. It uses screen-shots from a bespoke tool (the QuBE) developed by members of the LTT to combine responses from the departments and create the data files used by eXplorance Blue. As such the styling may differ from the final evaluations delivered to students.

The Anatomy of an Evaluation

There are some questions that will be standard to all evaluations delivered across the University. These occur in two places in the evaluation – at the start and at the end.
Here are the opening questions:

Three opening institutional questions

Departmental Questions

These are followed by up to three questions defined by each Department as common to their evaluations. These questions are drawn from a pool of 12 possible questions. Three possible selections are shown here:

3 department-wide likert questions

Module-specific Questions – Optional

The next block is optional and allows Departments to add up to three Likert-scale questions specific to each module being evaluated. These questions are set by the department – they are free text and so can be tailored to match the learning and teaching experience – e.g. focussing on tutorials, practicals, lab classes, field trips, etc. if they are used in this module. Some sample questions are shown here:

Sample module-specific questions.

 

Reflective Questions

All evaluations include two free-text questions allowing students to comment on positive and negative aspects of the module. These are delivered as separate questions to aid in the generation of reports.

Two reflective questions

Staff Rating – Optional

Some previous MEQs included a question where students could rate the quality of teaching delivered by staff on the module. In the review of MEQs many versions of this question were found. Some departments reviewed all staff (including tutors, demonstrators, etc.) others only lecturers, yet others no-one. The approach to staff rating was discussed at length at a recent Education Committee, noting the danger of linking this too closely to the probation process. The decision was made to allow an optional single repeating commonly worded staff rating question but that this should be restricted to lecturers only, to keep the size of the MEQ manageable. Departments are able to decide whether or not to include staff rating questions in their MEQs. If they choose to include such questions, they need to provide a list of the staff to rate for each module. An example is shown below:

Optional staff rating questions

Equality and Diversity

The questionnaires end with a final institution-wide question asking the student to reflect on their experience on this module. If students answer Yes to the question, they are prompted for more details on a separate page. Note that responses to this sensitive question are not included in the standard module report, they are only visible to Heads of Department and a few others.

Diversity and equality question

Compiling the MEQs

In this first year, we have been developing and rolling out the process as we go. We are very grateful for the timely help provided by staff in each department.

Data Gathering

The first step is to determine which modules are running and should be evaluated this year. The Academic Office has generated spreadsheets for each Department listing their modules and providing tools to allow them to select the questions to deploy at both the department and module level. These were then distributed to nominated individuals within each department who had the necessary knowledge and contacts to complete the forms.

Selecting departmental questions

Once returned, these spreadsheets are then fed into a bespoke system developed by members of the LTT to generate the feed files for use by eXplorance. Details of the nominated administrators, the teaching staff, the module leaders and students are obtained from relevant University systems. After checking, a series of feed files are generated and supplied securely to eXplorance.

Data Checking

To ensure that the correct modules are evaluated and the correct staff are rated, departments have a chance to preview the configuration before the system goes live. They are able to change the assignment of staff to be rated and the nominated module leaders. This process uses a special tool in eXplorance, known as DIG – the Data Information Gateway. Admin staff have been sent emails providing secure access to this  system.

DIG

Delivering the MEQs

Students eligible for MEQs will receive an email on the opening day of the evaluation period informing them that they have MEQs to complete and containing a link that allows them to complete them. They will also see the same list of MEQs available to them on a special MEQ tab in duo (handy if they later misplace the email):

A list of MEQs to complete, one per row. There is a completion graph on the right hand side.

The MEQ form itself is designed for display on mobile and desktop browsers. Reminder emails will be automatically sent to students who have yet to complete their evaluations as the deadline approaches.

Questions displaying nicely on a mobile phone

Staff

Module Leaders, Departmental Administrators and any staff whose teaching is being rated in the MEQ will see a list of the modules on the MEQ tab in duo:
Modules being evaluated are listed in a table, 5 at a time. At the RHS is a graph showing the completion percentage.

Staff will not be able to see any of the responses from this page, but if they click the name of any one of their evaluations they will get more details about the evaluation period and number of completions to date:
A page giving the module name, code, start and end date of the evaluation plus the number of students invited to complete a survey and the number who have actually responded. No names are visible.

Reports

Once the evaluation period is over, reports will be automatically generated for each evaluation. The exact nature of the reports is still being considered but we expect the list will be something like this:

  • Individual Staff Member Report – showing the results from the standard likert scale questions for the module, (plus the responses to this person’s staff rating question, if it is deployed).
  • Module-Wide Report – showing the results of all MEQ questions (except the equality and diversity question) this is available to the nominated Module Leader and Head of Department.
  • Equality & Diversity Report – showing the results of positive responses to this question and detail of the free text responses. This would only be made available to a very restricted set of individuals in the first instance – Heads of Departments and staff involved in supporting Equality and Diversity across the University.

Other aggregate reports are being developed (e.g. results across a department and programme). It has also been agreed to release reports for students, listing the results of the institutional and departmental likert questions (but not staff ratings, the equality and diversity question or any of the free text responses).

FAQs

FAQs re Module Evaluations

Below is a list of questions that may be asked about the new Module Evaluation process

Current Evaluation period

We are running Undergraduate end of module evaluations from Monday, 23rd April (2.00 pm) until midnight on Sunday, 6th May.  Reminders will be sent periodically throughout this period to those students still to complete evaluations.

Return Rates

The University requires a minimum 50% return rate for each end of of module evaluation, which is in line with return rates for the NSS (National Student Survey).

Students – How to Complete

Students will be sent notification emails (sent from evaluations@durham.ac.uk) which will include a link to a new system which lists the evaluations you need to complete.

The new evaluations have been made shorter and more consistent so should take less time to complete.  Evaluations can be completed on a pc or laptop, mobile or tablet device of your choice.

The University have decided on a focussed two week window, just before the exam period, in which we encourage students to take the time to give us their feedback.  Reminders will be sent to those students with surveys still to complete.

Students can also locate the evaluations they need to complete by clicking on the dedicated MEQ (Module Evaluation Questionnaire) tab in duo.  Students will also be able to see the percentage return rate.  Please help to increase the return rates to at least 50% by giving the University your feedback.

Staff – During the evaluation window

The surveys have been developed in conjunction with your Department/School and include questions on an institutional level, departmental level as well as a module level.  In some Departments/Schools a decision was made to include a core staff rating question and in others to exclude it for this year.  So this year you may or may not be rated.

If you login to duo you can locate the MEQ tab – this will allow you to view the evaluations list for the modules that you teach on or are module leader on.  Please view the Return Rate monitor for each evaluation.  Your involvement is vital in helping the evaluations to reach the required return rate.  When you meet with your students during the next few weeks please encourage them to take the time to complete their evaluations.  You may want to allow 10 minutes at the beginning or end of your session to allow the students to access their evalutions on a mobile or tablet – they can search for an email from evaluations@durham.ac.uk or locate the MEQ tab in duo.  You may want to explain the importance of giving feedback on the modules they have taken which will help the direction of teaching in the future.

Have a Question or a Problem?

If you have a question regarding the new Evaluation process or you have any problems completing surveys please email itservicedesk@durham.ac.uk and put “Explorance Blue” in the email subject field.

Reports

The different types of reports that will be available after the evaluation period and these will be located on the MEQ tab.

We will be developing not only reports for Teaching Staff, Module Leaders, Heads of Departments, Aggregate raw data reports for Administrators but also a report for Students.

Please note that we do not have a date for the release of the reports at this moment in time as we are currently developing them.

 

My MEQ report is grayed out saying Not Met?

The University (Curriculum, Learning & Assessment Service) has taken that for MEQs that receive less than 5 responses, that reports will not be run, in order to preserve the anonymity of the students providing responses.  So if you see the status Not Met – this means that it did not reach the threshold.

 

Why have I not been rated on my module?

As a member of staff looking for a staff rating response, please check that your Department/School asked for that question to be included.

If they did, ask them to confirm that your details were passed along.

Please note that if your involvement was as a tutor or seminar leader then you will not have been included in the process.

To keep the evaluations short we limited the staff rating question only to members of staff who were Lecturers and that the students will have been involved with.

Questions

If you have any questions about eXplorance Blue please email evaluations@durham.ac.uk and put “MEQ Advice” in the subject line.

Case Studies

Liverpool John Moores University amplifies the power of module evaluation data with Blue – READ

Alternatives

Alternative tools you might want to consider for interim evaluations include:

  • Blackboard Survey Tool – the course tool that sits in duo
  • Bristol Online Survey – the University has a site-license for staff and research students. Accounts are available by contacting the IT Service Desk. The data sits securely in the UK.

Please note that we do not advocate the use of other cloud based products that have not been cleared by the University – so please check with your Head of Department or the University Information Governance service before using anything not listed above.

REVIEW DATE: April 2018

PRODUCT: Blue 7