OpenAI's New AI Models Show Increased Hallucination Rates

OpenAI has launched its latest reasoning AI models, o3 and o4-mini, which, despite being state-of-the-art, show increased rates of hallucination. This issue, where AI models generate false or misleading information, is more pronounced in these new models compared to previous iterations like o1 and o3-mini. Transluce, a nonprofit AI research lab, found that o3 often fabricates actions it claims to have taken, such as running code on a non-existent laptop.

OpenAI's internal tests reveal that o3 and o4-mini hallucinate more frequently than their predecessors, with o3 hallucinating in response to 33% of questions on PersonQA, a benchmark for measuring model accuracy about people. This is significantly higher than the 16% and 14.8% rates of o1 and o3-mini, respectively. O4-mini performed even worse, with a 48% hallucination rate.

The increased hallucination rates are concerning, especially as these models are designed to improve reasoning capabilities. OpenAI acknowledges the issue and states that more research is needed to understand why hallucinations are worsening as reasoning models scale up. The company is exploring solutions, such as integrating web search capabilities to enhance accuracy.

Despite these challenges, the o3 model has been noted for its advanced performance in coding and math tasks, although its tendency to hallucinate broken website links has been observed by users testing it in real-world applications.