UCAT Scoring and Scaling: How does it work? Part 1
8 months ago by Rob
A common question among students preparing to sit the UCAT is how does UCAT scoring work? This blog answers that question. Practical implications for the UCAT test taker are covered in Part 2 of this blog.
Many UCAT companies claim that they can predict UCAT scores and percentiles from the number of UCAT questions that you answer correctly. This is unfortunately completely misleading and inaccurate. Most UCAT providers have little to no understanding of the scoring process.
In reality, UCAT scoring is extremely complicated. In this blog we will discuss how UCAT is scored and scaled.
How is UCAT scored and scaled?
Pearson VUE uses Item Response Theory (IRT or ‘Rasch modelling’) for UCAT scoring and scaling. This takes into account several factors such as the number of UCAT questions you got right (your raw UCAT score); the difficulty of UCAT questions; the Point Biserial Correlation Coefficient (PBCC); the probability that you obtained the right answer through random guessing; Differential Item Functioning (DIF); the cohort of students sitting the UCAT test, among other factors. To add to this complexity is the fact that you are randomly allocated one of 5 UCAT tests that Pearson VUE uses every year (this is to enhance the integrity of UCAT, reduce cheating and prevent ‘item leakage’).
While it is true that all questions carry equal marks, final UCAT scores take into account the difficulty of UCAT questions: so a student who answers 20 easy UCAT questions correctly will get a lower score than a student who answers 20 difficult UCAT questions correctly. That does not mean you should skip the ‘easy’ UCAT questions and focus only on the ‘hard’ UCAT questions, because scoring also takes into account the number of UCAT questions you get right.
How will you know what is a hard and easy UCAT question?
Furthermore, what is a difficult UCAT question for you may be easy for another candidate. However, there are UCAT questions which most students find generally easy or difficult. So how do you know which type of UCAT questions most students find easy or difficult? On MedEntry's online platform, you will obtain statistical information for each UCAT question, including the proportion of students who got the question right, the proportion of students who chose each option, how long other students spent on the UCAT question and so forth, which is extremely useful information. You should examine it carefully in your targeted UCAT preparation.
Because students of MedEntry are truly representative of the cohort who sit the UCAT (from all over Australia and NZ), you have a relevant and representative comparison group.
What is PBCC?
If a student who gets a high overall score in a UCAT subtest gets a question in that subtest wrong, it could mean there is something wrong with that question and its PBCC is low. Similarly, if a student who gets a low overall score in a UCAT subtest gets a particular question right, again there may be something wrong with the question and its PBCC is also low. UCAT aims for questions with high PBCC (at least 0.3). PBCC may be low because the UCAT question was poorly written, due to item ‘leakage’, or due to other factors. A question which previously had a high PBCC, which was used in the past and led UCAT testers to believe was a good question, may be deleted from scoring this year if its PBCC falls.
UCAT trial questions
UCAT test designers have a test bank of questions from which they randomly draw about 200 questions each year. They obviously want to keep adding to the UCAT test bank because of ‘leakage’ of some questions, the need to change UCAT question types and so forth. They therefore have UCAT question or ‘item’ writers who write questions regularly and consistently. These are then ‘tested’ to see if they are suitable by using them as ‘trial UCAT questions’ every year.
While the item writers have some idea of the suitability of the UCAT questions they created, they are not certain of their suitability unless they are tested on the ‘live UCAT cohort’ (for example, they may not know whether the PBCC is high enough). Approximately 10% of the total questions you will have in UCAT are these ‘trial questions’. Obviously, you will not know for certain which are trial UCAT questions (but you can guess: this will be discussed in the MedEntry workshop).
UCAT scoring software
Pearson VUE uses a software package for IRT calculations, which costs about $3,000 to buy. There are only a handful of such software providers in the world. To make things even more complex, there are certain assumptions the user of the package must input before the package can be used. Only experienced Psychometricians can do this. MedEntry purchases and uses these complex software packages for our UCAT scoring/scaling in order to ensure our UCAT exams and predicted UCAT scores are accurate.
Remember that UCAT scoring is extremely complex and is conducted by psychometricians. What is provided in this blog is a very brief and simplified summary. More details on how the UCAT scaling process works will be provided during the MedEntry UCAT workshops.
The blog UCAT Scoring and Scaling: How does it work? Part 2 discusses what the UCAT scoring and scaling process means for you in practice.