Cloudflare is testing a new way for websites to charge artificial intelligence companies that use their content to form models, perform inference or power search. The new functionality, called Pay to crawlIt allows websites owners to establish their conditions for how and when artificial intelligence robots can access their sites.
By default, Cloudflare now blocks artificial intelligence crawler from access to websites without permission. The owners of the site can choose whether to allow robots and decide whether access must be free or paid. Artificial intelligence companies, in turn, must reveal whether their crawler are used for training, inference or research, giving publishers more context to make decisions.
The idea is to help publishers resume some control over how their content is used in the ES economy. Many web sites are now scraped by models to without compensation, even if those same models remove traffic from traditional search engines.
In the last year, Cloudflare has launched tools to block or monitor artificial intelligence crawler. This includes a setting with a click to completely stop artificial intelligence robots and a dashboard that shows who is scraping what. Paying for scan is based on that work and puts a price on access, even if they are only a few cents per visit.
In private beta, publishers can choose to allow individual crawler, block them or fix rates for each scan. Cloudflare acts from medium, management of payments and forwarding revenues to websites owners. Both the publisher and the company needs Cloudflare accounts to participate.
The company states that the system could be useful in the future in which artificial intelligence agents – not people – visit websites, collect data and provide summaries or answers directly to users. In that scenario, users could give their agents to a budget to obtain high quality content from sources of trust. Cloudflare believes that a system such as scan salary can support that type of use.
There is a growing tension on how artificial intelligence companies use online content. Some big publishers like The New York Times They sued technological companies to scrape articles without permission. Others have signed license agreements to provide training data and allow their content to appear in the chatbot responses. But those agreements are limited to the main players and it is not clear how much they actually generate.
The smallest publishers have not had many options. The pay payroll could give them the opportunity to directly charge their content, rather than rely on the intermediaries or hope for reference traffic.
But convincing artificial intelligence companies will not be easy. Many of them are already scraping the content without permission. And the economy may not work in favor of the publishers, at least not yet.
Cloudflare has shared new numbers that show how the publishers rarely receive traffic from the Bot Ai. According to the company, Google’s crawler has scraped the sites 14 times for each click sent. Crawler of Openi has scraped 1,700 times for postponement and anthropic sites scraped 73,000 times for reference. For publishers who rely on traffic to support advertising revenue, it is a bad deal.
For years, publishers have let search engines like Google crawl their sites for free in exchange for visibility and click. That model is starting to break while the artificial intelligence tools replace the research, pulling the answers directly from the coveted content without sending users to the original source.
The default of Cloudflare for new websites is now to block the artificial intelligence crawler. The owners of the site must manually allow specific robots. The company calls this “default of control”.
A long list of media and content organizations expressed support for stronger protections on how their work is used, including Buzzfeed, Time, Reddit, Condé Nast and Stack Overflow.
If the Cloudflare model will work on a large scale it is still uncertain. It depends on the publishers who set the prices that make sense and on artificial intelligence companies are willing to pay. Cloudflare Matthew Prince’s CEO has also stated that the company is exploring the use of Stablecoins to support these transactions, creating precisely or collaborating with other suppliers.
It is likely that the agents to play a more important role in the way users access online information. In this case, the cloudflare system could be a first test of the fact that it is possible – or practical – to load the content that feeds them.
(Photo of John)
See also: Zuckerberg bets large on AI with $ 14 billion stakes and bad talents
Discover more from Gautam Kalal
Subscribe to get the latest posts sent to your email.
Be First to Comment