"We’re generating assessments faster than ever, but our real test is ensuring that these tools are fair and reliable across diverse candidate groups."–Louis HickmanIn this episode I welcome my friend, super dad, and ex- professional wrestler Louis Hickman for a killer conversation about the ins and outs of using LLMs to create and score assessments.Louis is a professor at Virginia Tech specializing in research on AI and large language models in assessment and hiring processes. He knows a thing or two about this stuff and we waste no time...
"We’re generating assessments faster than ever, but our real test is ensuring that these tools are fair and reliable across diverse candidate groups."
–Louis Hickman
In this episode I welcome my friend, super dad, and ex- professional wrestler Louis Hickman for a killer conversation about the ins and outs of using LLMs to create and score assessments.
Louis is a professor at Virginia Tech specializing in research on AI and large language models in assessment and hiring processes. He knows a thing or two about this stuff and we waste no time tackling some really great topics centering around the cutting edge of research and practice on the subject of LLMs and assessments.
This is a must listen episode for anyone developing, or considering developing, LLM based assessments. Or anyone who wants to educate themselves about how LLMs behave when asked to be I/O psychologists.
Topics Covered:
* LLMs in Assessment Center Role-Plays:
* Using LLMs to simulate realistic role-play scenarios for assessments, with the challenge of ensuring consistent, replicable candidate experiences.
* Evaluating Open-Ended Text with LLMs:
* How LLMs score open-ended responses and the observed biases, especially when diversity prompts only partially reduce disparities.
* Consistency in AI Scoring:
* Ensuring LLMs apply scoring criteria consistently across diverse candidates and settings.
* Applicant Reactions to AI Interviews:
* How candidates perceive AI-driven interviews, with many expressing discomfort due to the perceived inability to influence AI decisions compared to human interactions.
* Predicting Responses to Assessment Items:
* The potential for LLMs to predict candidate responses without actual data, though accuracy remains limited by model training and inherent biases.
* Impact on Academic Research:
* LLMs' influence on research publications, with concerns over AI tools favoring self-generated content and potentially amplifying biases in academic discourse.
Listen to the episode to hear the skinny on these topics and more!
And of course we have fun with this episode’s “Take it or Leave it” articles.
Article 1
“The Impact of Generative AI on Labor Market Matching.” An MIT Exploration of Generative AI”,
explores the use of LLMs on matching job seekers and employers.
Article 2
Four Singularities for Research: The Rise of AI is Creating Both Crisis and Opportunity
In this article from Ethan Mollick’s Substack blog One Useful Thing discusses the positive and negative impact of LLMs on academic research.
This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit charleshandler.substack.com
