Cloudflare Starts Blocking AI Crawlers by Default on Ad Pages This Week

Starting September 15, 2026, Cloudflare's default settings block AI crawlers that mix search, agent browsing and model training from any page that runs ads, unless the site owner opts back in. Cloudflare sits in front of roughly one in five websites on the internet. That's the number that makes this week's deadline matter far beyond one company's terms of service. Starting today, any new domain that signs up with Cloudflare, any new site added by an existing customer, and every customer still on the free tier will have a new default applied to their traffic: crawlers that can't or won't declare a single, clean purpose get blocked on any page carrying ads. Cloudflare calls these bots "multi-purpose crawlers," the kind that quietly do double or triple duty. Think of a bot that indexes a page for search results, feeds a chatbot's live answer, and hoovers up the same text to train a future model, all through one crawler that never tells the site which job it's doing in that moment. Under the new rules, Search stays allowed by default. Training and Agent access do not. And any crawler that blends the three gets judged by its most restrictive behavior. In practice, that means it's blocked. Cloudflare CEO Matthew Prince framed the shift as overdue given how the balance of web traffic has already tipped. "Now that the majority of traffic on the Internet is non-human, we must go further and act faster so that a sustainable ecosystem can emerge," he said, according to Cloudflare's own announcement. That's a stark shift. The company also published the numbers behind that urgency: bots recently overtook humans as the majority of internet traffic, and more than half of AI crawler requests are simply re-fetching pages that haven't changed since the last visit. Cloudflare singled out Google as an outlier too, saying its crawlers pull roughly twice as much information from the open web as those run by other AI companies. None of this happened overnight. Cloudflare first announced the policy on July 1, 2026, giving site owners and AI companies ten weeks to prepare before the September 15 cutoff arrived. That's a real runway. It also means any AI company still running an undifferentiated crawler this week chose not to fix it. Getting paid for the answer, not the fetch The blocking rules are only half the announcement. Cloudflare is also widening its Pay Per Crawl marketplace, which let publishers charge a bot for the act of scraping a page, into something it calls Pay Per Use. The distinction is the point: publishers now get paid when their content actually shapes an AI-generated answer, not merely when a bot happens to fetch the page. Two partners are live at launch. Ceramic.ai runs what it describes as a pay-per-query model, so a publisher gets paid each time their content shows up inside a Ceramic search result. You.com works differently, letting an AI agent pay on the spot when it needs a specific piece of premium content mid-conversation. Publishers who opt in also get reporting that tells them which queries surfaced their content, what snippet ran, and where they ranked, the kind of visibility that answer-engine optimization teams have been asking for since AI search started eating referral traffic. Condé Nast, Patreon and the newsletter platform beehiiv have all publicly backed the effort. That's not a small detail. It signals that publishers who've watched click-through traffic erode as chatbots answer questions directly, without ever sending a reader to the source, see Cloudflare's toll booth as a better deal than the status quo of getting scraped for free. Whether AI companies actually comply is the open question. Robots.txt has governed the truce between crawlers and website owners for roughly 30 years, and it has always been voluntary. Cloudflare's defaults carry more teeth because they sit at the network layer, not in a text file a bot can simply ignore. But a blocked crawler can still route around Cloudflare, spin up new IP ranges, or just eat the cost of being locked out of a fifth of the web's ad-supported pages. What's clear already is that Cloudflare has put itself in the middle of the transaction. For years, AI companies took web content largely for free and publishers had no practical way to say no. Cloudflare's scale gives that no some weight for the first time. Also read: Google Ships Gemini 3.8 Flash, Its Third Flash Model in Six Weeks • New York City Schools Ban AI Chatbots for 600,000 Young Students • CrowdStrike and the DOJ Dismantled the Sality Botnet That Stole Crypto for Years