OpenAI Introduces HealthBench for AI Model Evaluation in Healthcare

May 14, 2025
OpenAI has launched HealthBench, an open-source dataset designed to benchmark AI models in healthcare, featuring contributions from 262 physicians across 60 countries.
OpenAI Introduces HealthBench for AI Model Evaluation in Healthcare
Image: OpenAI

OpenAI has launched HealthBench, an open-source dataset aimed at benchmarking AI models in the healthcare sector. This initiative, detailed in a company blog post, involves collaboration with 262 physicians from 60 countries and includes 5,000 realistic health conversations. The dataset is designed to evaluate whether AI models provide optimal responses to health-related inquiries, using a physician-written rubric scored by GPT-4.1.

HealthBench's evaluation process highlights OpenAI's o3 reasoning model as the top performer with a score of 60%, followed by Elon Musk's Grok at 54% and Google's Gemini 2.5 Pro at 52%. The dataset supports 49 languages and covers 26 medical specialties, such as neurological surgery and ophthalmology.

An example scenario provided by OpenAI involves a 70-year-old unresponsive individual, where the AI model suggests steps like calling emergency services and checking airways. HealthBench scores the response, offering insights into the model's accuracy and areas for improvement, with a sample score of 77% for the scenario.

We hope you enjoyed this article

Consider subscribing to one of our newsletters like Life AI Weekly or Daily AI Brief.

Also, consider following us on social media:

Healthcare & Life Sciences AI AI Brief AI Brief (X)
Free newsletter

Life AI Weekly

Weekly coverage of AI applications in healthcare, drug development, biotechnology research, and genomics breakthroughs.

AI Token Webinar

Whitepaper

Tensordyne Napier: What If One Rack Could Do the Work of Nine?

Tensordyne

This Tensordyne whitepaper presents Napier, an inference-focused AI processor and rack-scale system based on the company’s TDN Math logarithmic number system. It examines infrastructure requirements for large mixture-of-experts and agentic models, compares major inference architecture approaches, and details the TDN AIP processor, TDN72 pod, TDN Link fabric, and Napier Ultra configuration. The paper reports simulation-based performance, cost, and accuracy-validation results, including Tensordyne’s projected comparison of one Napier rack with a nine-rack Nvidia Rubin plus Groq deployment; the chip is reported as taped out and in fabrication.

Read more

You may also like

Free, six days a week

Daily AI Brief: the AI news that matters, in your inbox.