How can medical education assessment harness recent advances in AI?

NLP Data Scientist, Victoria Yaneva, discusses NBME’s approach to innovation in AI solutions for assessment, driving research excellence, and promoting transparency through collaboration and active community engagement.

Natural Language Processing (NLP), a subfield of Artificial Intelligence (AI), is reshaping the landscape of medical education assessment by offering innovative avenues to evaluate essential skills. In collaboration with fellow scientists, my team conducts extensive research to develop NLP and AI capabilities tailored to the unique needs of the field. This post highlights the core tenets of our approach to innovation: driving research excellence and promoting transparency through collaboration and active community engagement.

AI Capabilities Tailored to the Needs of Medical Education Assessment

To evaluate complex constructs like communication and clinical reasoning, medical educators require innovative assessments that go beyond the multiple-choice format. However, such assessments require the ability to efficiently score a vast number of free-text responses or deliver personalized feedback to large learner groups within tight timelines. My team specializes in developing NLP/AI capabilities, including technology-assisted scoring, improved content creation, and automated provision of personalized feedback. Recent applications include a system for scoring responses to open-ended items, predicting item difficulty based on text, and facilitating clinical item writing through automated distractor suggestion.

Progress and Transparency Through Community Engagement

In our pursuit of developing specialized AI solutions, we recognize the dual imperative for rapid progress and responsible, transparent AI research practices. To navigate this balance, we lean into community engagement by openly sharing data and extending invitations for NLP/AI researchers to contribute in creating open-access solutions available for public scrutiny and evaluation. We also bring together researchers across AI, assessment, and medical education to form a shared understanding of key challenges. Examples of such initiatives include:

By fostering community engagement in AI for medical education, we uphold transparency and excellence in research, while cultivating a diverse group of researchers to propel advancements in critical areas. As an institution that values partnership, we share innovations publicly, allowing other organizations to build upon these solutions. Together, we are shaping the future of AI in medical education assessment.

Medical education needs to support the advancement of skills and behaviors alongside knowledge, so students can develop as complete physicians, ready to take on patient care. We’re rethinking measurement to facilitate this evolution, but we can’t do it without new perspectives and ideas.