AI chatbots may prove more effective than traditional search engines in countering foreign propaganda, according to a recent study conducted by NPR in collaboration with NewsGuard, an organization specializing in assessing online misinformation. With the rise of AI tools like chatbots and Google's AI-driven answers, there have been growing concerns about the potential manipulation of these technologies by governments aiming to disseminate misleading narratives.
NPR's investigation revealed that while AI chatbots performed admirably in rebutting state-sponsored falsehoods, AI-generated summaries at the top of search engine results didn’t perform as well, though they still managed to debunk misleading information most of the time. Mike Caulfield, a digital literacy expert at the University of Washington, Bothell, emphasized that utilizing AI chatbots alongside traditional web searches can be beneficial for users exploring complex topics.
The experiment aimed to evaluate how well various AI tools and search engines handle questions based on false premises. Researchers crafted 30 questions that centered on disinformation propagated by countries such as China, Iran, and Russia. These questions were directed at major AI chatbots, including OpenAI's ChatGPT and Google's Gemini, as well as to leading search engines. The results showed that AI chatbots successfully debunked about 75% of the false narratives presented to them, surpassing traditional search engine performance.
For instance, when asked about a false claim regarding a Ukrainian monastery bombed by Russia, the chatbots accurately identified the misinformation, noting that it originated from a Russian disinformation campaign. Traditional search tools, while not entirely ineffective, failed to challenge misleading claims more frequently.
Importantly, NPR found that AI chatbots were less likely to repeat false claims uncritically compared to search engines, where misinformation can proliferate unchecked in some cases. The analysis also revealed that while AI-generated summaries from search engines do debunk false narratives to some extent, they are generally less reliable than direct chatbot responses.
Performance varied among the AI outputs from different search engines. Google's AI Overview performed better than Microsoft Bing's summaries, while DuckDuckGo’s results fell in between. All three search engines provided AI summaries at varying frequencies, with Google consistently offering them for almost all queries tested.
Experts advise a careful approach when utilizing AI tools for research since the quality of information can be significantly affected by the sources used. Wack, a digital persuasion researcher, highlights that traditional search engines have always had their share of imperfections, reinforcing the idea that users must remain discerning regardless of the format of information access.
The research underscores the idea that a combined strategy using both AI chatbots and traditional search engines may offer a more nuanced approach for users navigating misinformation. While AI chatbots show promise in providing context and credibility analysis for sources, tracking down the original sources remains critical for validating claims. Therefore, whether leveraging traditional search engines or AI-generated content, a thorough examination of the underlying information is essential to ensure accuracy in an increasingly complex digital landscape.
