Introduction
So, one of the questions I get asked all the time is, “Could you mark some more of my homework questions?”
We also do loads of homework questions during the course, including homework questions that we mark. We also do a mock exam and an exam prep. So, I think we probably do about 100 questions, but there are about 70 learning outcomes in the whole course.
Given that there are four different types of questions, you’re not going to do each type of question for each learning outcome. So, I can understand how people want more questions and more marking.
Introducing the AI Tool
We can now programme an AI tool to generate and mark questions. We’ve been working on this for 18 months. So I just thought I’d make this brief video showing you what that can do, its strengths, and its limitations at the moment.
So basically, there are four types of questions in the exam:
- Multiple-choice – A one-mark question with several answer options.
- Select from list – Two missing words to be selected, worth a total of 2 marks.
- Short response – Two words or short phrases, worth a total of 2 marks.
- Long response – A short essay, typically five sentences long, worth 5 marks.
I think this tool is most useful for those long response questions. The exam is a 2 1/2-hour assessment. There are 90 marks available, and those long response questions make up fifty of those 90 marks. So the long answer questions are by far the most significant. They’re also the hardest for you to self-evaluate and self-mark; this is where this AI tool is really helpful.
Demonstrating the Tool
This is where the power of this AI tool is instrumental in helping you practise and develop the skills to respond to those long response questions. So here’s a typical question you might have for a long response question.
Explain three benefits of coaching and mentoring and explain 2 benefits of the role of emotional intelligence in implementing coaching and mentoring within the team.
So, this is what a typical answer might look like. The APM gave us this one part of the answer:
Skills development: By adopting a coaching and mentoring approach, team members can receive personalised guidance and feedback.
So it’s a fact plus a supporting statement — the what and the why.
So, how can we practise this? I’m going to show you this tool and how it works. If you come to our course, we will share this URL with you so you can access this experimental APM PMQ chatbot.
It’s programmed with the syllabus, the study guide, all the course material, and everything else. I will ask it to generate a question like the one we just looked at from the APM.
So, it generates a scenario about managing a newly formed team from various departments, using coaching and mentoring and developing emotional intelligence. The question it generates is:
Explain two benefits of coaching and mentoring and explain three benefits of emotional intelligence in leading a team.
Answering and Marking
I’ve pre-prepared an answer. The first part is the APM’s typical answer—skills development. Then, I’ve written a second part, improving delegation. For emotional intelligence, I’ve written about regulating your feelings, empathising with others, and understanding yourself.
Let’s see what it thinks of that answer. It clearly likes the first one because that was from the APM. It also likes the others and gives full marks, with some areas for improvement: make it more specific and link it back to the question. It also provides a reference from the Parallel study with a guide to help you read more.
Marker Moderation and Accuracy
Now, how do we know that these marks are correct? What have we done on marker moderation?
We routinely conduct a marker moderation exercise. Trainers are given questions and the APM marking guide and asked to grade them. We compare their grades and analyse the data statistically.
We did this with 10 markers and the tool. GPT was marker 94. In most cases, GPT was very close to the human markers. Sometimes, it was a little more generous or stricter but within standard deviation.
Based on these moderation results, the marks generated by the tool are fairly similar to those produced by a human trainer. We then fed this data into the model to improve its marking ability.
Evaluating a Weaker Answer
Next, we looked at a weaker question:
Describe 2 reasons why you would need to conduct regular re-estimating of the project schedule and cost.
Candidate answer:
- Re-estimating helps confirm a realistic scope and timeline.
- Helps us understand the options.
But this doesn’t draw on the scenario, which mentioned supply chain issues and the design phase. Also, understanding options isn’t a clear benefit of re-estimating.
Another part of the answer said:
- Highlight roles and responsibilities.
That’s not really a benefit of re-estimating. Better examples would be enhanced risk management and better resource allocation.
The tool gave 2.5 marks and offered an improved version with better alignment to the question.
Multiple Choice Questions
Now, let’s look at some multiple-choice questions. People say the real exam’s multiple-choice questions are trickier than the ones here. That’s because the AI is trained to write plain English, which isn’t ideal for making them tricky.
We tried a question on proactive risk responses. The tool generated this:
Which of the following are proactive measures to reduce threat?
Options included: Avoid, Accept, Transfer, Reduce, Monitor, Defer.
Correct proactive answers were: Avoid, Transfer, and Reduce. The tool accepted that answer and confirmed it.
Just a word of caution — these are a bit easier than the real ones, but we’re working to improve that. Please send feedback if you find them too easy or hard.
Using the Tool for Coaching
You can also use the chatbot as a coaching tool. If you’re confused about configuration management or earned value, ask the tool to explain these topics to you.
For example: Explain threat reduction measures.
It will summarise the content from the study guide — e.g., avoid, reduce, transfer, reactive responses, etc. You can ask for examples. It’s more interactive for those who prefer that learning style.
Conclusion
I hope you find this helpful tool. A good strategy is to practice long answer questions, get feedback, and improve your technique.
Remember, people say that language in the multiple-choice questions might be more straightforward than the real ones, but you have many other multiple-choice questions to practice.
If you find any errors or hallucinations, please let me know. I can tweak and reprogram the tool.
We would be grateful for any feedback, positive or constructive. We’re only 18 months into this journey, and who knows where it will lead?
Thank you for your time. I hope you find it useful. Good luck with your exam!
