I had #OpenAI's ChatGPT take a USMLE 119 question exam. It scored 70% I removed questions that depend on an image. Questions have 4-6 choices.

Millionmeal @million_meal

3 years ago

I had #OpenAI's ChatGPT take a USMLE 119 question exam. It scored 70% I removed questions that depend on an image. Questions have 4-6 choices. https://t.co/Q56pNtlzxD

0 0 4 0 0

@pythonprimes Wow very cool!

@pythonprimes Can you try it on Step 2? It's supposed to be the more clinically relevant one usmle.org/sites/default/…

@pythonprimes Wow. Is this Step 1 or Step 2. Just out of curiousity. If Step 1, I believe that 70% correct may be a Pass, and since it is Pass/Fail exam that definitely says something. Step 2 is more about diagnosis/treatment, I wonder if ChatGPT would also perform like that or perhaps better.

@pythonprimes 70% on both? That’s … convenient… 🤔

@pythonprimes Which exam questions did you use? I haven't been able to get chatgpt to score nearly as high as 70% on sample Step2 questions from Uworld (the most used practice question bank). The question you posted looks to be from Step1 if I had to guess and I haven't tried those yet

@pythonprimes Very cool. What percentile would this be?

@pythonprimes I wonder how it’d fare on the image questions using this technique:

K Sadov @cherrvak

3 years ago

@pythonprimes I wonder how it’d fare on the image questions using this technique:

1 0 5 551 1