eXplorance Blue Course Evaluations

Explorance Blue Logo

The University has changed its approach to module and programme evaluation and has adopted a new tool to deliver it. This approach is being driven by the PVC (Education). The initial implementation project was led by Dr Tony Fawcett (Dean of Education), Andrew Rennie (Head of the Curriculum, Learning and Assessment Service), Dr Malcolm Murray (eLearning Manager) and Julie Mulvey (eXplorance Expert). It draws on the findings of the MEQ Task & Finish Group led by Prof Gordon Love. Where decisions are required they have been obtained from the University Education Committee. Data Transfer has been approved by the University Information Governance Unit.

In the past, the University has used a range of tools to deliver module evaluations, most recently Blackboard’s Enterprise Survey Tool. During 2016-17, the University ran a successful year-long pilot of eXplorance Blue with Durham Business School. At the same time, Education Committee ran a Task & Finish Group looking at the module evaluation process. The result was a desire to produce a common set of questions at the level of the University and then within a Department. This will allow aggregation of the data and better reporting. To this end, in May 2017, the University signed a 5 year deal with the Canadian company Explorance Inc. They are a leading provider of course/module evaluation software with clients from across the globe.

Implementation Timetable

The Durham Evaluation Unit began in 2017-18 with end of year undergraduate module evaluations. The  eXplorance Blue module evaluation solution has been made available to  all departments during the Easter term of 2018 and this tool should be used for all evaluations. After completing Undergradute MEQs, the project will then expand to cover Postgraduate MEQs and programme-level evaluations. Staff who need to create evaluations for interim module evaluations or modules that finish early in the academic year) should use Blackboard’s survey tool and are being advised not the Enterprise survey tool.  Once eXplorance Blue takes over the MEQ process fully, the Enterprise Survey tool in Blackboard will become read-only.

Enterprise Surveys:  Please note that are are not training any new Evaluation Managers for the Enterprise Survey tool and recommend that if you are still using it then you only allow staff who know how to use the tool to have access.  Module Codes can be added by sending them as a Service Desk ticket via the IT Service Desk.

Blackboard Survey tool:  This is a tool that sits within each duo modules and allows a survey to be build and run anonymously.  You can copy a survey template from one course into others and instructions can be found on the Blackboard Help area within the module.

The New Approach

Structured Questions

Standard questions have been defined to provide a more consistent student experience and allow aggregation of the data (so results from modules with low numbers of students can still contribute to a broader, programme-level picture). The question style and responses have been designed to follow those used in the NSS. There was also an acknowledgement that some existing questionnaires were too long, risking a low return rate. Some existing MEQs contained questions that should really be asked earlier in interim evaluations when there is still time to make changes for the current cohort. As such the number of questions has been limited, so that evaluations should be a similar length across the University.

Evaluations will be driven by the data in the student record system. As such we will be evaluating modules by module code, not duo course codes or any other way of aggregating students.

Evaluations are made up of distinct sets of questions. The explanation that follows focuses on the MEQ process. It also applies to programme evaluations – simply replace the word ‘Module’ with ‘Programme’ in the examples below. It uses screen-shots from a bespoke tool (the QuBE) developed by members of the LTT to combine responses from the departments and create the data files used by eXplorance Blue. As such the styling may differ from the final evaluations delivered to students.

The Anatomy of an Evaluation

There are some questions that will be standard to all evaluations delivered across the University. These occur in two places in the evaluation – at the start and at the end.
Here are the opening core institutional questions:

Three opening institutional questions

Departmental Questions

These are followed by up to three questions defined by each Department as common to their evaluations. These questions are drawn from a pool of 12 possible questions. Three possible selections are shown here:

3 department-wide likert questions

Module-specific Questions – Optional

The next block is optional and allows Departments to add up to three Likert-scale questions specific to each module being evaluated. These questions are set by the department – they are free text and so can be tailored to match the learning and teaching experience – e.g. focussing on tutorials, practicals, lab classes, field trips, etc. if they are used in this module. Some sample questions are shown here:

Sample module-specific questions.

 

Reflective Questions

All evaluations include two free-text questions allowing students to comment on positive and negative aspects of the module. These are delivered as separate questions to aid in the generation of reports.

Two reflective questions

Staff Rating – Optional

Some previous MEQs included a question where students could rate the quality of teaching delivered by staff on the module. In the review of MEQs many versions of this question were found. Some departments reviewed all staff (including tutors, demonstrators, etc.) others only lecturers, yet others no-one. The approach to staff rating was discussed at length at a recent Education Committee, noting the danger of linking this too closely to the probation process. The decision was made to allow an optional single repeating commonly worded staff rating question but that this should be restricted to lecturers only, to keep the size of the MEQ manageable. Departments are able to decide whether or not to include staff rating questions in their MEQs. If they choose to include such questions, they need to provide a list of the staff to rate for each module. An example is shown below:

Optional staff rating questions

During the academic year 2018/19 the staff rating question was expanded to include Lecturers, Support Teaching Staff or where it is a dissertation module to include a list of Supervisors.  The lecturer questions are mandatory, however the Support Teaching Staff / Supervisor roles are optional and students are asked to select from a list.

Equality and Diversity

The questionnaires end with a final institution-wide question asking the student to reflect on their experience on this module. If students answer Yes to the question, they are prompted for more details on a separate page. Note that responses to this sensitive question are not included in the standard module report, they are only visible to Heads of Department and a few others.

Diversity and equality question

Compiling the MEQs

Data Gathering

The first step is to determine which modules are running and should be evaluated this year. The CLAS Unit has generated spreadsheets for each Department listing their modules and providing tools to allow them to select the questions to deploy at both the department and module level. These were then distributed to nominated individuals within each department who had the necessary knowledge and contacts to complete the forms.

Selecting departmental questions

Once returned, these spreadsheets are then fed into the Blue “Data Integrity Gateway” for double checking by two nominated individuals from each department.

Data Checking

To ensure that the correct modules are evaluated and the correct staff are rated, departments have a chance to preview the configuration before the system goes live. They are able to change the assignment of staff to be rated and what roles they have, ie Lecturer, Support / Teacher or Supervisor and the nominated module leaders (full report viewers). This process uses a special tool in eXplorance, known as DIG – the Data Information Gateway. Admin staff have been sent emails providing secure access to this  system.

DIG

Delivering the MEQs

Students eligible for MEQs will receive an email on the opening day of the evaluation period informing them that they have MEQs to complete and containing a link that allows them to complete them. They will also see the same list of MEQs available to them on a special MEQ tab in duo (handy if they later misplace the email):

A list of MEQs to complete, one per row. There is a completion graph on the right hand side.

The MEQ form itself is designed for display on mobile and desktop browsers. Reminder emails will be automatically sent to students who have yet to complete their evaluations as the deadline approaches.

Questions displaying nicely on a mobile phone

Staff

Module Leaders, Departmental Administrators and any staff whose teaching is being rated in the MEQ will see a list of the modules on the MEQ tab in duo:
Modules being evaluated are listed in a table, 5 at a time. At the RHS is a graph showing the completion percentage.

Staff will not be able to see any of the responses from this page, but if they click the name of any one of their evaluations they will get more details about the evaluation period and number of completions to date:
A page giving the module name, code, start and end date of the evaluation plus the number of students invited to complete a survey and the number who have actually responded. No names are visible.

Students

It is very important for you to know the following:

  • The required Return Rate for all modules is 50% and you play a part in helping the University achieve this by giving us your feedback
  • Please note that the response threshold is set at 5 ensuring then a report will not be run in if it does not meet the threshold in order order to maintain the anonymity of the students, however the quantitative data will be rolled into aggregate reports at a departmental level
  • Please refrain from making any personal comments on individual members of teaching staff. All students are expected to comply with the University’s “Respect at Work” and “Respect at Study” policies and any inappropriate comments will be removed before viewing by teaching staff (www.dur.ac.uk/equality.diversity/positiveworking/policies)
  • Data from the Discrimination question will be disseminated to the Head of the Curriculum, Assessment and Learning Unit (CLAS) in order to speak to Departments concerned.

Reports

Once the evaluation period is over, reports are automatically generated for each evaluation. The reports are as follows:

  • Full Module Report – showing the results of ALL MEQ questions (apart from the equality and diversity question) this is available to the nominated Module Leader and Head of Department.  It is the responsibility of the Module Leader to interpret the data and feed back to the individual teachers the results of the free text comments (this is in order to check if there are any inappropriate personal comments which will be removed).
  • Individual Teacher Report – showing the results from the standard likert scale questions for the module, (plus the responses to this person’s staff rating question, if it is deployed) – but no free text responses.
  • Equality & Diversity Report – showing the anonymous results of positive responses to this question and detail of the free text responses. This would only be made available to the head of the Curriculum, Learning & Assessment Unit (CLAS) who will analyse the responses and ensure that issues are addressed.
  • Departmental Spreadsheet Report – shows all the modules and all the questions with all the corresponding ratings listed – no free text responses.

FAQs

FAQs re Module Evaluations

Below is a list of questions that may be asked about the new Module Evaluation process

Current Evaluation period

We are running Undergraduate end of module evaluations from Monday, 23rd April (2.00 pm) until midnight on Sunday, 6th May.  Reminders will be sent periodically throughout this period to those students still to complete evaluations.

Return Rates

The University requires a minimum 50% return rate for each end of of module evaluation, which is in line with return rates for the NSS (National Student Survey).

Students – How to Complete

Students will be sent notification emails (sent from evaluations@durham.ac.uk) which will include a link to a new system which lists the evaluations you need to complete.

The new evaluations have been made shorter and more consistent so should take less time to complete.  Evaluations can be completed on a pc or laptop, mobile or tablet device of your choice.

The University have decided on a focussed two week window, just before the exam period, in which we encourage students to take the time to give us their feedback.  Reminders will be sent to those students with surveys still to complete.

Students can also locate the evaluations they need to complete by clicking on the dedicated MEQ (Module Evaluation Questionnaire) tab in duo.  Students will also be able to see the percentage return rate.  Please help to increase the return rates to at least 50% by giving the University your feedback.

Staff – During the evaluation window

The surveys have been developed in conjunction with your Department/School and include questions on an institutional level, departmental level as well as a module level.  In some Departments/Schools a decision was made to include a core staff rating question and in others to exclude it for this year.  So this year you may or may not be rated.

If you login to duo you can locate the MEQ tab – this will allow you to view the evaluations list for the modules that you teach on or are module leader on.  Please view the Return Rate monitor for each evaluation.  Your involvement is vital in helping the evaluations to reach the required return rate.  When you meet with your students during the next few weeks please encourage them to take the time to complete their evaluations.  You may want to allow 10 minutes at the beginning or end of your session to allow the students to access their evalutions on a mobile or tablet – they can search for an email from evaluations@durham.ac.uk or locate the MEQ tab in duo.  You may want to explain the importance of giving feedback on the modules they have taken which will help the direction of teaching in the future.

Have a Question or a Problem?

If you have a question regarding the new Evaluation process or you have any problems completing surveys please email itservicedesk@durham.ac.uk and put “Explorance Blue” in the email subject field.

Reports

The different types of reports that will be available after the evaluation period and these will be located on the MEQ tab.

We will be developing not only reports for Teaching Staff, Module Leaders, Heads of Departments, Aggregate raw data reports for Administrators but also a report for Students.

Please note that we do not have a date for the release of the reports at this moment in time as we are currently developing them.

 

My MEQ report is grayed out saying Not Met?

The University (Curriculum, Learning & Assessment Service) has taken that for MEQs that receive less than 5 responses, that reports will not be run, in order to preserve the anonymity of the students providing responses.  So if you see the status Not Met – this means that it did not reach the threshold.

 

Why have I not been rated on my module?

As a member of staff looking for a staff rating response, please check that your Department/School asked for that question to be included.

If they did, ask them to confirm that your details were passed along.

Please note that if your involvement was as a tutor or seminar leader then you will not have been included in the process.

To keep the evaluations short we limited the staff rating question only to members of staff who were Lecturers and that the students will have been involved with.

Questions

If you have any questions about eXplorance Blue please email evaluations@durham.ac.uk and put “MEQ Advice” in the subject line.

Case Studies

Liverpool John Moores University amplifies the power of module evaluation data with Blue – READ

Alternatives

Alternative tools you might want to consider for interim evaluations include:

  • Blackboard Survey Tool – the course tool that sits in duo
  • Bristol Online Survey – the University has a site-license for staff and research students. Accounts are available by contacting the IT Service Desk. The data sits securely in the UK.

Please note that we do not advocate the use of other cloud based products that have not been cleared by the University – so please check with your Head of Department or the University Information Governance service before using anything not listed above.

REVIEW DATE: April 2018

PRODUCT: Blue 7