Confusion between information and memes, wrong dates, fabricated details… AI apps such as ChatGPT make mistakes almost once in two when asked about current affairs, according to a major study released today by European public radio and television networks.

This test was done on four artificial intelligence applications that are widely used in the world for information: the ChatGPT (of the OpenAI company), the Copilot (Microsoft), the Gemini (Google) and the Perplexity.

Overall, 45% of their responses contained “at least one major problem”, regardless of language and region, according to the study coordinated by the European Broadcasting Union (UER, EBU) and based on previous BBC work.

It even contained an answer to the five “major mistakes“, with outdated information, even with “contrived details».

Of the four applications, Gemini had the worst results: there were “significant problems” in three-quarters of its responses, which is “more than twice as many as in the other apps”, mainly due to its “moderate performance” in correctly citing the source of a piece of information.

Examples

Twenty-two public media from eighteen countries, mostly European, took part in the study. From late May to early June, every media outlet asked the same questions about the current state of AI applications, asking them to look for the answers in their content.

Out of 3,000 responses, outdated information was one of the most common problems.

In the question “Who is the pope?“, the answer given by the ChatGPT in the Finnish public media group Yle, as well as by Copilot and Gemini in the Dutch media NOS and NPO it was “the Francis». However, Francis had already passed away and his successor, Leo XIV, had already taken office.

Asked by the French group Radio France about the alleged Elon Musk’s Nazi salute during the inauguration of Donald Trump in January in the United States, the Gemini wrote that the billionaire had “right arm erection”. The AI ​​application in question had taken literally a satirical vignette by humorist Charlene Vanhenaker.

“AI assistants are still not a reliable way to consume information,” Jean-Philippe De Tender, deputy director-general of the UER, and Pete Archer, head of artificial intelligence at the BBC, conclude in the study.

However, these AI applications are increasingly being used for information, especially by young people. According to a global report, published in June by the Reuters Institute, 15% of under-25s use them every week to get a brief update on the news.

The study, released today, is “the first of its kind,” Mathieu Beauval, Radio France’s director of innovation, told AFP.