Shocking 2023 Examiner's Report on Question 4 Paper 1
The Examiner’s Report for 2023 makes disturbing reading, for me anyway.
The Chief Examiner says:
Students who answer the reading paper questions in a different order, got lower grades in 2023.
This is because, to answer Question 4, students can reuse information that they used in answering questions 2 and 3.
Students who did Question 4 before Questions 2 and 3 got worse grades in 2023.
These are presented as FACTS.
I know from supporting many schools, and watching students in exams, that many students simply lose energy or motivation toward the end of the exam. So, for many I have recommended doing the paper in the ‘wrong order’: question 5 first, and next question 4.
I tell students to try this in their mocks or a practice paper, to see if this works for them.
ONLY do it if it works for you.
But, what if my advice is damaging students in the real exam? I’ll have to change it.
So, this is what I wrote to AQA several weeks ago:
My Email to AQA
Hello Team AQA English,
Thank you for the examiner's report to the 2023 exams. As always, it is so useful in knowing how best to prepare our students.
However, I am confused by some of the advice about question 4 of Paper 1.
"Some students appear to be responding to the questions in an unorthodox order. Section A is designed to build the skills for Question 4, the highest tariff question in that section. Although timed conditions place certain demands on students, those students who have taken the steps through Questions 1-3 tended to have more insight into writer’s methods, as well as more thorough answers for Question 4."
I completely understand the rationale at the beginning of this statement, but what evidence did you use in stating the conclusion in bold? As a school that does alter the question order, this is very pertinent to us.
I also ask this because the statement appears to contradict further comments about question 4:
"Short answers which meant that the discussion could only be short and the ending of the extract was not included.
Answers seemed to be shorter than in previous years, despite the significant number of details to discuss and the accessibility of the question. An answer can come in at any level, but brief responses inhibit movement up the levels."
We would expect students answering question 4 first to make time to write long answers to question 4 (and your comments about exam timings also imply this).
You point out that students taking this approach wrote less well on all the questions. However, this appears to be contradicted by this statement:
"Students who seemed less confident on Questions 2 and 3 often wrote more successful answers for Question 4."
You also note that:
"Although the question itself engaged students and offered many opportunities to discuss the statement, students seemed less inclined to draw across the skills from Questions 2 and 3. Exploring language features and/or discussing the structural shifts are credited for this question."
So this implies that in reality students did not actually benefit from having answered questions 2 and 3 first, even though that is the intent of the exam.
In short, my questions can be summed up with just this:
What numerical data did you use to decide that students who did question 4 first (or varied the question order) performed less well?
(E.g. We sampled 500 candidates, 200 hundred did question 4 before question 3 and 2.
When we analysed the two groups, 300 candidates following the chronological order averaged 5 marks on question 2, 5 marks on question 3 and 12 marks on question 4. In contrast the corresponding marks for those doing a different question order were: 5, 4, and 10 respectively".)
We also ask our students to answer question 5 first. Do you have any data on how successful this approach is?
Thank you,
D. Salles
I don’t want to publish their answer, as it is a private correspondence. So, I will summarise it. I am happy for any examiner to read this, because everything I am about to tell you is entirely supported by that reply.
Summary of AQA Reply
We do not know if any student answered the questions in a different order. It is impossible to tell.
We made up those facts about question order, and the result on students’ marks. We have no evidence for what we said.
This is because we didn’t do anything to test our made up facts, to see if they are likely to be true.
We made this all up because of one real fact. The real fact is that the average mark for question 4 went down this year.
The only possible explanation for this is that they have been getting some very bad advice from you, Mr Salles, and other online influencers.
Well, here are some alternative reasons:
The question was harder!
I can’t give away what it was about, but you needed to know about a particular animal, not native to the UK or Europe.
You needed to know about some of its typical behaviours in the wild. Play around on Tik Tok and see how students reacted on the day! It is funny, if you happen to know all about this animal.
Could that be a reason that the answer was shorter?
Exam boards are supposed to control for this sort of knowledge, and make sure their questions don’t depend on it.
Example
If I ask you to read and understand an extract about the Pope, Christian students are likely to know a lot more than non-Christian students. Catholic Christians are likely to know a lot more than other Christians (because the Pope is Catholic*).
The results of the question will not be a fair test of students’ ability to read and interpret, because many of the marks will depend on a lot of prior knowledge.
This is why you don’t get questions on the Pope, or Formula 1 racing, or politics, or the Roman Empire, or 1000 other things where some groups of students might know much more than other students.
The pandemic has changed student behaviour.
There has been a large increase in students giving up earlier - which we can clearly see in the rise of students being absent from school. (The rate of absence has nearly doubled since pre-pandemic 2019)
Might it be logical to assume that students are more inclined to give up with the harder, longer questions at the end of the exam paper?
I think so.
I’ll be publishing a top grade answer to question 4 in my next post for paid subscriber later today.
You’ll see how much you have to write, and how difficult it is to do well in this question without effort, without giving up.
*Very few of you will understand this line: ‘does the Pope shit in the woods?’
You will need to know that this is a reference to two common cliches:
Is the Pope a Catholic? This is the answer to any question where the answer is obvious.
E.g. Student: do I have to underline the title and date, sir? Teacher: Is the Pope a Catholic?
The other cliche can also be given in reply: “Do bears shit in the woods?”
This makes the reply “does the Pope shit in the woods?” funny because you know it is intended to mean, “yes, of course, stop asking me dumb questions”, while at the same time saying something that we are pretty sure is not true, and therefore leading to the opposite answer, “no, don’t underline the title and date”.
See what I mean? Where you have no insider knowledge, it is very hard to work out what is going on.
(For the record, I have never responded to a stupid question from a student with this reply, because they know very little about the toileting habits of Popes, and it is never a good idea to use the word ‘shit’ in public).
Ooops. Being polite about AQA is causing my anger to slip out in other ways.