Introduction
From the first weeks of the pandemic, it was clear that the usual face to face exams were not an option for the June 2020 call. Thus, the rectorate had to decide whether to postpone them or to implement a remote examination system. Since face-to-face exams represent an important stake for the quality of UNED's teaching and learning processes, this decision was crucial and had to be taken with urgency. After exploring the available market solutions, the rectorate decided to implement its own software to allow for remote exam sitting. This software is called Aula Virtual de Exámenes – AvEx ("virtual examination hall").
While trying to replicate the usual conditions of face-to-face exams, fraud prevention was an essential requisite, and legal certainty was also to be guaranteed. Facial recognition-based solutions were discarded in favour of a multilevel approach which succeeded in enforcing the University's strict rules for examination. Facial recognition and its problems In recent years, FRT applied to surveillance is expanding. However, despite the promises of sellers of these AI-based technologies, there is mounting evidence that these technologies can cause serious harm and that there are technical, legal, and ethical difficulties associated with them. UNED, as a pioneer institution in distance learning, which recently welcomed its community participation to develop an ethical framework for the use of massive data-based technologies (UNED, 2019), pondered a key question regarding remote examination: can we delegate the responsibility to guaranteeing cleanness in online exams to technology only? In order to find an answer, a series of potentially problematic issues was identified (for a detailed account of these issues, see Aznarte et al. 2021):
a/ There is no clear legal framework for invasive surveillance technologies.
b/ The use of FRT might imply a violation of the legal principles of necessity and proportionality.
c/ FRT can violate privacy rights.
d/ FRT are naturally imprecise, and its software is fallible (Simonite, 2019).
e/ FRT can produce automatization bias (Lyell, 2016).
f/ FRT can produce discriminations and violations of the equality principle.
g/ FRT can generate discriminations based on different functional abilities.
Last, but not least, it is a known fact that FRT, in spite of their promises, are unable to guarantee authorship or to avoid the use of non-permitted materials in online tests. As hackers have taught us, it is almost impossible to conceive a system which is 100% secure. There is no way to avoid someone having two keyboards and screens connected to the same computer, for example, or accessing the exams from a virtual operating system. These are only two of many possible creative examples.
AvEx security measures
Given the above, UNED decided to implement AvEx, a system without FRT, relying on a set of incomplete measures that, together, guarantee the fairness of the examination process. This software allows for synchronous and simultaneous examination of thousands of students and tries to replicate the conditions of face-to-face tests without TRF. AvEx security measures can be divided into three different categories as follows.
Security elements related to test design
In remote examination, one of the fundamental factors to reduce fraud is to be found in the very design of the exam. This must be balanced with the fact that fraud minimization is not the only objective: any change to the tests should not imply an increase in their difficulty. Some of these measures are:
- Wide question banks: combinatorial automatic generation of exams allows for a drastic reduction of the probability of two students having the same exam. In AvEx, these banks can be created by course section, thus allowing for a balanced exam to be offered to every student.
- Open book exams: teams were encouraged to allow the use of any material while taking the examination. In any regard, non-memoristic questions are considered to be a better tool for evaluation, and thus it is a good opportunity to shift our evaluation culture.
- Time/number of questions ratio: Adjusting the allotted time for the exam, depending on its complexity, implies that less time is available for cheating. Teachers were encouraged to also take into account low-bandwidth situations.
Software related security elements
Since UNED developed its own software, and despite the urgency of the first weeks, was able to introduce several security elements from the very first version of AvEx:
- Randomization of questions and answers: since exams are taken synchronously by the students of a single course, randomization impairs knowledge sharing.
- Sequential access to questions: difficulting the accumulation of questions/answers and thus the capturing and sharing of them.
- Disabilitation of certain software freedoms: impossibility to copy and paste text from the application hinders quick sharing of information.
- Verification: UNED's own text matching engine, allowing for the automatic comparison of students' answers, both amongst them and with other texts either from the course material or from the internet. Coincidences can thus be highlighted and the course responsible can act upon them.
- Random picture taking: during the test, students are required to allow access to their webcam, and photos are taken randomly. These pictures are not automatically analysed, but they are attached to the exam and thus available for the course teams to verify authorship.
- Remote assistance (under development): AvEx will allow bidirectional audio-visual communication between a student and a teacher in charge of the process. This communication can be used both for solving procedural questions and for verifying and preventing any unwanted behaviour.
Other security measures
The University put other exceptional measures in place to guarantee the fairness of the process. One example is a sworn declaration of authorship and of no violation of the regulations: Students are informed about the existing sanctioning system while an appeal to their individual responsibility is made. This signed document will be used in eventual legal processes. Another is a delayed interview: Students agree to the possibility that the course responsible fix an appointment with them, as a complement to the exam, in order to confirm the result of the evaluation.
Conclusion
It is important to consider the evaluation system as a whole: it is the decisive conjunction of the different aforementioned elements that helps fraud minimization. This way, we can guarantee rigour and quality of our evaluation, without the need of using overrated and problematic technologies in the process. The results of AvEx after five examination calls from June 2020, with a low number of issues recorded and a generally high student and teacher satisfaction, are encouraging and draw a clear path for the future.
