What is ‘Question Level Analysis’?

An article by Richard James Rogers (author of The Quick Guide to Classroom Management).

Dr. Curry was a friendly, but direct, man.

On one sunny afternoon in the newly-built science labs at St. Richard Gwyn RC High School, Flint, he called me to the back of the class to discuss my progress in the subject. This was a strategy that was being implemented school-wide, with every ‘A2’ – Level student (we were in our final year of high school).

Giving feedback

Dr. Curry must have made me think that day because I remember what he said:

“You’re still waffling too much in exam questions, Richard, but it’s better than when you were doing GCSEs”

I hope I won’t waffle too much in this blog post.

Thought results in memory 

Daniel Willingham sums up what I experienced best in possibly one of the most influential texts on pedagogy out there: ‘Why Don’t Students Like School?’

Memory is the residue of thought

This quote often comes up in my thoughts as I go about my day as a teacher. I think about the thousands of lessons I was taught as a kid in primary and secondary school. In terms of content, I think I can remember upwards of 70% (as proven by my exam scores over the years). In terms of impressions and actually thinking back to a specific lesson and ‘remembering’ what happened: that’s got to be less than 1%.

img_0009-1
An AMAZING book!

Dr Curry’s analysis got me thinking about myself, and how I had evolved as a student over the years. It was a good tip, and it helped, but is there a way to make a one-to-one meeting like this even more specific (and is it beneficial)?

Command terms frequency analysis

Whole-class feedback can be powerful, especially when there are obvious patterns of weakness showing-up. This is where an early-technique of mine can be really helpful: analyzing the command terms in different tests and assessments.

Discussing homework

Command terms are those words that tell you what to do in a question. Examples include:

  • Describe
  • Annotate
  • Explain

What I found helpful in the early part of my career was counting the command terms used in different tests, and then seeing which students scored poorly in tests where a clear skew of command terms was apparent. This gave me some means of specific advice that I could pass on to my learners:

If we look at the table above, for example, we can see that the different tests contain different numbers of command terms. Let’s say that I have a student: ‘Student A’, and Student A does okay in test 1 in November; as expected in the December test, but slumps in the January test. What does a teacher do in that situation and why has this happened?

As we can see from the frequency analysis above, it may not be the case that Student A simply didn’t revise enough (a common misconception that teachers have).

chatting in class

We see that the January test required many more explanations and calculations that any of Student A’s previous tests, and her poor score could just be because she hasn’t had enough practice in completing these kinds of questions.

Armed with this kind of data, teachers can have very meaningful and powerful one-to-one conversations with students: highlighting specific areas of weakness and providing guidance on how to best tackle specific command terms.

This process can be a little tedious, however, and focuses only on skills (not content).

So……..is there a way to make this process better?

Thankfully, the answer is yes!: it’s called Question Level Analysis or QLA.

Students assess, think, share and discuss: productive QLA

Taking the command terms frequency analysis to the next level (which is a great ‘whole-class snapshot tool’) we can now get the students to analyse their own responses to specific questions and then think carefully about how to fix any problems they’ve had (creating memory in the process).

jenga

The method I like to use (because of its simplicity and ‘hands-on’ approach for the students) is as follows:

  1. Share an editable spreadsheet with all of your students (such as a Google Sheet®). Make sure the command terms are filled in for each column heading.
  2. Get your students to fill in their names and colour in the boxes with a colour that recognizes levels of understanding (e.g. green, amber and red)
  3. You could do this as a ‘per test’ format or cumulatively over the course of a year, examination course or even the entirety of a child’s schooling:

4. Pair the students up (or group them) according to weakness-matching. For example, student B can help student G with the calculations questions, and student G can help student B with the ‘describe’ questions.

5. Once the kids are paired up, give them the chance to explain to each other how to answer the specific questions that they scored poorly on. Ideally, this should be done before any mark-scheme or model answer sheet is given to the students, as this will cause deep-thinking to take place, which should result in secure memory of the concepts being learned in the process.

poll-everywhere

6. Rotate students around a few times: this will get those who’ve just learned something new to teach it to another student (you’ll have to explain this concept to the class)

6. Once the process has finished, feel free to give the mark-schemes to the students: they’ll need them when doing their own revision before the final exams. Sometimes students make messy notes when they are peer-teaching each other like this, so be sure to tell them to keep their corrections tidy and clear as they’ll need to refer to them as part of their revision at some point in the future.

Q & A

Teacher actions during the QLA process of peer-teaching

  • Sit-in on different pairs of students and listen to how the conversations are going
  • Call individual pairs to your desk and ask probing questions about areas of weakness
  • Mark questions in front of the students in cases where complete misunderstanding has taken place
  • Sit with very weak students and provide extra-guidance
  • A possible twist: pair some very strong students up and get them to create a website, blog or infographic that teaches all of the students the content in the exam

I’ve been fortunate enough to use QLA successfully in my teaching over the years, and I saw it in action at Harrow International School at a CPD course I went on last weekend. I can tell you firmly and confidently: it works!

img_0262
Discussing QLA with Harrow International School staff last week 

IMG_5938

richard-rogers-online

We welcome you to join the Richard Rogers online community. Like our Facebook page and follow us on Twitter for daily updates.

Latest hybrid

Author:

High School Science and Mathematics Teacher, Author and Blogger. Graduated from Bangor University with a BSc (Hons) degree in Molecular Biology and a PGCE in Secondary Science Education. Richard also holds the coveted Certificate in Mathematics from the Open University (UK).

2 thoughts on “What is ‘Question Level Analysis’?

Leave a Reply