ChatGPT and Google Gemini are terrible at summarizing news, according to a
new study

Date:
Tue, 11 Feb 2025 12:00:05 +0000

Description:
A new study finds AI chatbots more often than not inaccurately summarize news.

FULL STORY

A new study from the BBC has found that four of the world's most popular AI chatbots including ChatGPT are inaccurately summarizing news stories.

The BBC asked ChatGPT , Copilot, Gemini , and Perplexity to summarize 100
news stories from the news outlet and then rated each answer to determine
just how accurate the AI responses were.

The study found that "51% of all AI answers to questions about the news were judged to have significant issues of some form." and "19% of AI answers which cited BBC content introduced factual errors, such as incorrect factual statements, numbers and dates."

The study showcases multiple examples of inaccuracies that showcased
differing information to the news it was summarizing. The examples note that "Gemini incorrectly said the NHS did not recommend vaping as an aid to quit smoking" and "ChatGPT and Copilot said Rishi Sunak and Nicola Sturgeon were still in office even after they had left."

Inaccuracies aside, there's another crucial finding. The report found that AI "struggled to differentiate between opinion and fact, editorialised, and
often failed to include essential context."

While these results are unsurprising considering how often we see issues with news summarization tools at the moment, including Apple Intelligence's
mix-ups that have led Apple to temporarily remove the feature in iOS 18.3 , it's a good reminder not to believe everything you read from AI.

Are you surprised?

From the study, the BBC concludes that "Microsoft's Copilot and Google's
Gemini had more significant issues than OpenAI's ChatGPT and Perplexity,"

While this research doesn't necessarily give us much more info, it validates the skepticism towards AI summary tools and emphasizes just how important it
is to take information from AI chatbots with a pinch of salt. AI is
developing rapidly and large language models (LLMs) are released almost
weekly at the moment, so it's to be expected that mistakes will happen. That said, from my personal testing I've found inaccuracies and hallucinations to
be less frequent now in software like ChatGPT than it was just a few months ago.

Sam Altman said in a blog post yesterday that AI is progressing faster than Moores law and that means we'll continue to see constant improvements to software and how it interacts with the world around it. For now, however,
it's probably best not to trust AI for your daily news.

======================================================================
Link to news story: https://www.techradar.com/computing/artificial-intelligence/chatgpt-and-google -gemini-are-terrible-at-summarizing-news-according-to-a-new-study

$$
--- SBBSecho 3.20-Linux
* Origin: capitolcityonline.net * Telnet/SSH:2022/HTTP (1:2320/105)

ChatGPT and Google Gemini are terrible at summarizing news, according to a >MP> new study

No one should be shocked by this. So called "AI" is just your phone's autocorrect on steroids.

No one sensible should be shocked. How many people do you know who would likely take AI's word for it, so long as the misinformation it provides confirms their beliefs?


* SLMR 2.1a * If it's Tourist Season, howcum we can't shoot 'em, Pa?
--- SBBSecho 3.20-Linux
* Origin: capitolcityonline.net * Telnet/SSH:2022/HTTP (1:2320/105)

ChatGPT and Google Gemini are terrible at summarizing news, according to a new study

No one should be shocked by this. So called "AI" is just your phone's autocorrect on steroids.


--- AfterShock/Android 1.7.5
* Origin: South of Heaven - Chaos rampant, an age of distrust (1:154/30.1)

ChatGPT and Google Gemini are terrible at summarizing news, according new study

No one should be shocked by this. So called "AI" is just your phone's autocorrect on steroids.

Are you not buying into the hype of AI?

I haven't seen it do anything too awesome yet. Gemini has been a little handy for getting examples of things, but I haven't seen it save any lives yet, or take anyones job.

I have a difficult time believing anything until I see it, and I haven't seen AI yet.

--- Mystic BBS v1.12 A49 2023/04/30 (Windows/64)
* Origin: JoesBBS.Com, Telnet:23 SSH:22 HTTP:80 (1:342/201)