Impact of AI Search Engines on Publishers

A recent report from the content licensing platform TollBit, shared with Forbes exclusively, highlights a sharp decline in referral traffic from AI-enhanced search engines to news sites and publishers. The findings indicate that AI-driven search tools, including those created by OpenAI and Perplexity, deliver 96% less traffic to publishers in comparison to conventional Google searches.

While AI companies assert that their innovations will open up new revenue avenues for publishers, the evidence suggests the contrary. In recent months, AI developers have intensified their web scraping efforts, with entities like OpenAI, Perplexity, and Meta scraping a collective average of two million websites in just the fourth quarter of 2024. The research, which examined 160 websites spanning national and local news, tech consumer content, and shopping blogs, revealed that each page was scraped approximately seven times on average.

Toshit Panigrahi, the CEO of TollBit, emphasised the extent of this data extraction. He noted that there has been an upsurge in bots targeting these platforms whenever a user raises a query. He remarked that the demand for publisher content is significant.

Effects of AI Search Engines on Publishers

Nathan Schultz, CEO of Edtech firm Chegg, has expressed concerns regarding the dangers that AI search engines present to businesses reliant on search traffic. He stated that it is crucial to refuse these new dynamics, suggesting that the traditional equilibrium between content creators and search engines is being disrupted.

Chegg’s experience acts as a cautionary tale. In January 2025, its website traffic experienced a drastic drop of 49% year-over-year, starkly contrasting with an 8% decline in Q2 2024, shortly after Google rolled out its AI summaries. This steep fall has prompted Chegg to contemplate going private or seeking acquisition.

The report from TollBit found that AI firms collectively scraped sites around two million times each quarter in 2024. However, these scrapes seldom result in traffic being directed back to the publishers. With each page being scraped an average of seven times, this becomes an added burden on server costs without generating extra income.

Challenges Linked to Opaque Scraping Practices

A significant hurdle for publishers is the opacity surrounding AI web crawlers. AI companies employ a variety of user agents to harvest data, yet many fail to adequately disclose their activities. Olivia Joslin, a co-founder of TollBit, highlighted the predicament faced by publishers, noting that blocking Google could have adverse effects on SEO. She mentioned the difficulty in discerning the specific intentions behind the AI bots’ web activities.

Perplexity, currently valued at $9 billion, has come under fire for its scraping strategies. Even when publishers try to restrict its access, the company continues to drive referral traffic, which raises concerns about undisclosed scraping methods. The report revealed that Perplexity scraped a particular publisher’s site 500 times while sending over 10,000 referrals, raising questions about the strategies employed to source the content.

This AI firm has previously received backlash for redistributing paywalled articles from major publishers like Forbes, CNBC, and Bloomberg without sufficient attribution. Additionally, it has displayed AI-generated blogs and subpar content in search results. CEO Aravind Srinivas acknowledged challenges with the republishing function, describing it as an ongoing project.

Exploring Pathways for Publishers

To counteract these challenges, some publishers are pursuing licensing arrangements with AI enterprises to secure fair remuneration. Notably, The Associated Press, Axel Springer, and The Financial Times have established content agreements with OpenAI. Furthermore, companies like TollBit have adopted models that charge AI firms for each scraping instance.

If this situation remains unregulated, analysts warn of an approaching scenario termed “AI slurry”, where the integrity of quality journalism is jeopardised, pushing reputable content creators out of business and inundating the internet with inferior information.

The Road Ahead for AI and Search

AI-powered search engines are likely here to stay. As an increasing number of users gravitate towards AI chatbots and research tools rather than traditional search engines, it is essential for the industry to strike a balance between innovation and equitable compensation for content creators. Without clearly defined frameworks and enforceable licensing agreements, the future of digital journalism and the quality of content available online are both left in jeopardy.