New Cloudflare feature lets websites charge AI developers for content access

By Maria Deutscher for SiliconANGLE
Wednesday, July 2, 2025

The pay per crawl feature uses a public key cryptography algorithm called Ed25519. It encrypts data by encoding it into mathematical structures known as twisted Edwards curves that were discovered in 2008. Besides Cloudflare’s AI crawler verification mechanism, the algorithm is also used in the popular SSH protocol for logging into servers and other cybersecurity tools.

“Given that Cloudflare protects the majority of the world’s most popular websites, as well as millions of smaller websites that publish academic and scientific content, this security feature will elegantly prevent data-greedy bots from unwarrantedly scraping human-created content without permission and without paying for it,” said Ilia Kolochenko, the Chief Executive Officer of cybersecurity company ImmuniWeb.

Cloudflare debuted the pay per crawl feature alongside several other new AI-focused capabilities. Going forward, the company’s CDN will block AI crawlers from accessing websites by default. There’s also a new setting that allows publishers to only block crawlers from accessing pages with ads. Read Full Article

Previous Media Publications:

Intelligent CISO: Cloudflare introduces permission-based internet scraping for AI crawlers, signalling a new business model

Infosecurity Magazine: Cloudflare Now Blocks AI Web Scraping by Default