Earlier in our series on medical exams and AI we suggested it was not enough simply to focus on ever more sophisticated barriers to the use of AI in exams.
Yes, there are various actions that can be employed in connection with the actual physical and digital environment that the exams are taken in. For example, for on-site digital exams, enforcing strict bans on all personal digital devices (such as smart watches, glasses, and so on); for remote digital exams, a focus on multi-layered monitoring software and use of a digital exam platform that locks the examinee out of the rest of the laptop.
However, a more sustainable approach lies in re-designing the assessment questions so that the question formats, and the questions themselves, are structured in such a way that they are not easily answered by medical AI tools. This includes the use of multi-stage, progressively revealed patient cases; use of un-labelled raw media, such as pathology slides or ECG tracings; and complementing digital tests with short, live viva components or OSCEs ((Objective Structured Clinical Examinations).
In this article we take a deeper dive into some of these ideas and how they might be applied to make vignette questions more resistant to the use of AI.
AI resistant vignette questions
To make clinical vignettes resistant to AI, it’s necessary to move away from standard textbook case structures. LLMs excel at processing linear, predictable patterns of symptoms to find a textbook diagnosis. Changing the way information is delivered, and what the student is asked to do with it, can help to address this vulnerability.
1. Progressive Disclosure (Linear Forward Testing)
Standard vignettes give all the patient’s data upfront, allowing an AI to process the entire case instantly. Instead, structure the digital exam so the student must answer Question A before they can unlock Question B.
• This prevents students from backtracking. If Question A asks for initial triage steps based only on vital signs, they cannot use the lab results revealed in Question B to reverse-engineer the answer.
2. Multi-Layered Information Noise
AI handles clean, concise data perfectly. By contrast, real patients present with irrelevant history, non-specific symptoms, and conflicting data.
Reflect the reality of clinical practice by injecting ‘clinical noise’ into the vignette. Include the patient’s irrelevant chronic conditions, complex social histories, non-contributory family medical backgrounds, and vague, distracting symptoms.
• LLMs often over-analyze minor details and are distracted by red herrings, leading them to jump to diagnosing rare conditions. A human student must use clinical judgment to filter out the noise and focus on the acute problem.
3. Media-Dependent Questions
Text-based AI cannot analyze raw, non-standard visual data under strict exam timers.
Accordingly, do not describe the physical exam or lab findings in text. Instead, embed an unlabelled image of a pathology slide, a unique ECG strip, a specific dermatological lesion, or an ultrasound video clip.
• The question might read: “Based on the attached image and the patient’s vitals, what is the next best step?” If the text description is missing, the student cannot simply copy-paste text into a standard AI prompt. Even multimodal AIs struggle with low-resolution or heavily contextual clinical images.
4. Test “Next Steps” Over Diagnosis
AI is highly accurate at naming a disease based on a list of symptoms, but it struggles with the nuanced logistics of hospital care. Therefore, shift the question focus away from “What is the diagnosis?” and towards local protocol, resource management, and communication.
• Ask questions like:
◦ “Which of these four orders should the nurse execute first?”
◦ “How would you explain this poor prognosis to the family given the cultural context provided?”
◦ “Given that the main CT scanner is down today, what is your alternative diagnostic pathway?”
5. Localized Guidelines and Formula Restrictions
AIs are trained on global, generalized medical knowledge, which often defaults to specific national standards (like USMLE guidelines).
Instead, explicitly state that the question must be answered according to your specific local hospital trust protocols, regional health guidelines, or a very specific, newly updated medical board paper.
• If the AI defaults to standard national or international guidelines, but your exam grades are strictly based on local regional protocols, the AI-generated answer will be marked incorrect.
Putting it into Practice: A Before and After Example
Here is a before-and-after example of a standard medical vignette rewritten to make reliance on AI difficult. The original question relies on textbook patterns that an LLM can solve in seconds. The revised version introduces visual dependencies, clinical noise, and localized constraints that force the student to apply true clinical reasoning.
The Before Example (Vulnerable to AI)
Vignette: A 62-year-old male presents to the Emergency Department with crushing substernal chest pain, radiating to his left arm for the past 45 minutes, accompanied by diaphoresis and nausea. He has a history of hypertension and hypercholesterolemia. His vital signs are: BP 145/90 mmHg, HR 98 bpm, RR 18 rpm, Temp 37°C. An ECG shows ST-segment elevation in leads II, III, and aVF.
Question: What is the most likely diagnosis? a) Acute Pericarditis b) Inferior Wall Myocardial Infarction c) Aortic Dissection d) Pulmonary Embolism
❌ Why AI defeats this easily:
• Textbook Pattern: The text lists a flawless, classic presentation of an inferior STEMI.
• Keywords: Words like “crushing substernal chest pain”, “radiating to left arm”, and “ST-segment elevation in II, III, aVF” act as immediate triggers for the AI’s training data.
• Simple Recall: It only asks for a diagnosis, which is the easiest task for a medical AI.
The After Example (AI-Resistant)
Vignette: A 62-year-old male is brought to your local ED by his partner. He complains of general chest discomfort, but spends the first few minutes of the interview anxious about his missed flight to Spain and his history of gastroesophageal reflux disease (GERD), for which he ran out of omeprazole two days ago. He also mentions mild osteoarthritis in his left shoulder that has been “flaring up” all morning.
His vitals are stable, but he looks pale and clammy. The nurse hands you the printed ECG telemetry strip ([Image Embedded: A custom-cropped, unlabelled ECG showing dynamic ST elevation in II, III, aVF with reciprocal depressions in I and aVL]).
Question 1 (Progressive Disclosure – Student cannot see Question 2 yet): Based on the attached image and the local clinical pathway guidelines for your specific regional health trust, which of the following immediate management steps must be initiated first? a) Administer a trial dose of an oral proton-pump inhibitor. b) Request an urgent chest X-ray to rule out aortic pathology before giving antiplatelets. c) Activate the primary percutaneous coronary intervention (PCI) pathway and administer loading doses of aspirin and a P2Y12 inhibitor as per local Trust formulary. d) Order a CT Pulmonary Angiogram due to high risk from his upcoming flight history.
Why this beats the AI:
• Clinical Noise: The AI gets bogged down by distractions. The missed flight hints at a pulmonary embolism (d), the left shoulder pain hints at musculoskeletal issues, and the omeprazole history hints at GI issues (a). A human clinician knows to prioritize the pale/clammy appearance and look at the ECG.
• Media-Dependent: The text does not mention ST-segment elevation. If the student copies and pastes the text into an AI, the AI lacks the critical piece of data (the ECG image) to make the diagnosis. It will most likely guess a GI issue or PE based on the text.
• Localized Logistics: Options c and d require knowing specific local hospital protocols (“Trust formulary”), rather than generic global guidelines.
• Next-Step Focus: It skips asking for the diagnosis entirely. It forces the student to look at the image, diagnose the patient implicitly, and choose the correct logistical management action.