Summative use of Questionmark Perception

The Questionmark Perception system provides many features designed specifically for the running of on-line examinations (summative assessment). However careful consideration of a number of issues is necessary when implementing this type of use, and you should seek approval from the Chair of your school Learning & Teaching Committee before using on-line assessment for summative testing of students.

For high stakes secure exams it is very important to involve the Questionmark support staff in developing the scheduling and arrangements, to maximise server performance and reliability.

First some terminology - summative assessment types

Low stakes summative assessment - Performance in the assessment is used to assign a small proportion (<10%) of module marks.  Students can generally take the assessment any number of times, at anytime and anywhere with access to any other resources.  Marks are allocated based on performance in the assessment.  This can be taken from the average over multiple attempts, the best mark over multiple attempts or the mark for the final attempt. 

Medium stakes summative assessment - Performance in the assessment is used to assign a significant proportion (10-30%) of module marks.  Students can generally take a trial form of the assessment prior to the invigilated, timetabled assessment under exam conditions.  Marks are allocated based on performance in this invigilated assessment.  Access to other resources during the exam is restricted through using Questionmark Secure for delivery.   

High stakes summative assessment - Performance in the assessment is used to assign a very significant proportion (40-100%) of module marks.  Students can generally take a trial form of the assessment prior to the invigilated, timetabled assessment under exam conditions.  Marks are allocated based on performance in this invigilated assessment.  Access to other resources during the exam is restricted through using Questionmark Secure for delivery.   

The following guidelines apply to all summative assessments delivered on-line

Accessibility

Duty to ensure equitable access to assessments means that where possible accessible question types and components (in particular images, multimedia, etc.) are used.  The examiner must be open to offering flexible time periods and alternative versions of assessments where necessary. Considerations should include:

1. Providing dyslexic students with additional time (as this also applies to computer based tests);
2. Providing users of screen readers with an alternative assessment with an appropriate time allocated;
3. Using alternative question types to those that involve fine motor control (e.g. drag and drop questions) for students with physical disabilities.
4. Avoiding question types that require a high cognitive load where this is not pedagogically important (e.g. ranking and multiple response questions with large numbers of options).

Students are requested to register disabilities with their school, so your school student support team should be able to identify any individuals who may require special consideration.
Allowing students to attempt trial versions of an assessment prior to the exam is very important in highlighting where accessibility issues might arise (as well as familiarising the students with the interface, etc.) and gives time for alternative provision to be made.

Choice of assessment template

Good practice and student feedback suggests that the question by question templates (e.g. qxqon) are most appropriate for use in summative exams.  The important functionality offered by these includes the ability to:

1. return to previously answered questions during the exam;
2. removes the requirement to scroll to navigate through the questions;
3. flag tricky questions;
4. get an overview of which questions have been answered;
5. provide a prompt on submission if not all questions have been answered.

N.B. - Question by question templates are difficult to use with a screen reader and do not provide the same benefits for this type of user.  The default or Questionmark template with all questions delivered in one question block is more appropriate in this case.

Participants & Scheduling

• Unique participant IDs (ISS usernames) and randomly generated passwords should generally be used to provide access to summative assessments via a schedule and perception.dll.
• Spare logins set up in advance for invigilated summative exams are very useful in emergencies.
• All participant access to the assessment will be via participant log-in and assessment scheduling (open access to the assessment should be prevented on both the staging and production systems by unchecking the appropriate option when authoring the assessment).
• The period of access should be limited in the schedule, to prevent access before or after the exam.
• Limiting the number of attempts should be used with care, as technical problems may occur with an initial attempt.

Quality assurance of assessment questions

Questions used as the basis for assigning marks to a module should be subject to the standard quality assurance procedures such as review by external examiners.  On-line questions might be better presented in a paper based form for this process.
The on-line assessment itself should also be tested prior to running the exam by both the assessment author and appropriate teaching peers.

Procedures to maximise server reliability

• All summative assessments should be delivered on the production system.
• Use of the reporting tools on the production system must be prevented during an exam.
• Staggering of student start and submission of the assessment should be used to avoid overloading the production system, especially for large cohorts.
• Close communication with ISS and Questionmark support staff is essential so that they are aware of when such exams are being run.

Security

• Medium and high stakes assessments should be invigilated, as with paper based examinations, to check student identities and prevent inappropriate activities during the exam.
• Ultimate security can be provided by checking the assessment Requires monitoring setting (open the assessment editor, double click on the Control block and go to the Security settings ). This requires a Questionmark administrator to log-in before the student can take the assessment (this approach may be problematic for large cohorts).
• The layout of PC clusters used for exams should discourage cheating (i.e. each students' view of other PC screens minimised);
• Use Questionmark Secure (selected in the assessment security settings) to prevent access to the Internet or other software packages where this is important (N.B. contact the Questionmark support team if you want to use this, to check that the cluster you are using has Questionmark Secure installed);
• It is good practice to get students to complete and sign a form in the exam which includes their name, the login ID they have used and the id of the PC on which they sat the exam;
• If student cohorts are large then staggered groups need to be kept apart and communication between them prevented (remember mobile phones!);
• Use random question ordering and random ordering of options in each question to reduce the chances of copying;
• The selection of a random subset of questions from a large bank of questions is considered good practice, but care must be taken to ensure the difficulty of the exam is consistent for all participants;
• Prevent access to the summative assessment via open.dll both when testing on the staging system and on the live system;
• Ensure scheduling prevents access to the assessment before and after the examination period.
• Display of marks and question feedback should be turned off in the assessment settings unless specifically desired.

Student familiarity with Questionmark

The purpose of examining using Questionmark is not to test a student's ability to use the assessment interface.  Therefore for summative applications students must be given an opportunity to become familiar with this tool in a non-examination environment.
Part of this familiarisation process is developing an understanding of:

1. the question types and scoring allocation (particularly if negative marking is used);
2. use of the interface and navigation;
3. the time allowed and how this is controlled (this can be controlled within Questionmark).

So providing formative assessments with similar question types and the same user interface prior to a summative assessment is good practice.

Negative marking

Negative marking can be applied using the Questionmark authoring tools.  If you intend to use this then ensure that 'Don't know' options are available without score penalties and that students clearly understand how marks are to be allocated for the assessment.
One suggested negative marking scheme is to use negative fractional marks for wrong answers (e.g. -0.25 for MCQs with 5 options) so that an assessment attempt that involves random choices would receive zero total marks (rather than 20 out of 100 for MCQs with 5 options), but note Questionmark can only provide whole number marks (i.e. an MCQ with 5 options receives -1 for a wrong answer and +4 for a correct answer).

Support during the exam

In general make sure the Questionmark support team can attend the set up prior to and the beginning of the exam. 
A communication channel between the assessment invigilator and Questionmark support team should be available throughout the exam.

After the exam

A back-up of the exam marks must be made shortly after the exam has been completed (e.g. request the Questionmark support team generate an Export to ASCII report of all marks and store this securely).

Page owner: help@vles.leeds.ac.uk