Cloudflare's 1% Pilot Feeds ChatGPT Search [2026]
Cloudflare and OpenAI say a research test involving about 1% of Cloudflare’s network helped ChatGPT Search find and add hundreds of millions of web pages to its search index that it had never found before. This information comes from Search Engine Journal’s report on the October 15 results. The Cloudflare and OpenAI pilot began in July 2026. It is still a research project, not a product available to everyone.
Cloudflare also made two other changes during the same three-week period. On October 2, it opened a Web Search API for testing. The service launched with three outside providers whose listed prices differ by a factor of 28. In mid-October, Cloudflare also announced that it would begin checking Verified Bots’ behavior over time. These checks include whether bots follow robots.txt rules and slow down after receiving an HTTP 429 response.
Together, these changes show where Cloudflare wants to fit into the growing AI search market: between the bots that read websites and the websites themselves. It wants to help AI companies find useful pages while giving website owners more control over how bots access their sites. Cloudflare has already worked with OpenAI to fix security problems at the network edge. The search pilot builds on that existing relationship.
What Cloudflare and OpenAI Reported on October 15
The announcement appeared in a Cloudflare blog post written with OpenAI staff. Cloudflare first described the research on July 8. It then ran the test with a group that represented about 1% of its entire network. By October 15, the companies said the information collected had helped OpenAI’s search crawler find and index hundreds of millions of web pages that it had not known about before. These pages could now be included in ChatGPT Search.
The companies did not give an exact total. They only said that hundreds of millions of pages had been found. Other coverage of the announcement uses the same wording, and no news outlet has reported a more precise number. This matters because Cloudflare says its network reaches locations within 50 milliseconds of 95% of the world’s internet-connected population. Even 1% of that large network covers many websites.
The pilot had a specific goal. OpenAI wanted to find out whether information from across a network could help its crawler spend more time on pages worth adding to its search index. Cloudflare wanted to show that the information it sees about web traffic could help without giving OpenAI the actual contents of web pages. The companies also said they used web metadata to help with search and discovery, rather than sharing page content for training AI models.
The examples in the announcement were ordinary but useful pages: a guide for a particular washing machine, the latest passport rules and fees, and a restaurant’s menu and opening hours. Cloudflare’s point is that these pages can be very helpful to people even when few other websites link to them. A crawler that finds pages by following links from pages it already knows can easily miss them.
How the Discovery Signals Work
Cloudflare sits between many websites and their visitors. Because of this, it can see which pages get real visits, which pages have changed, and which web addresses do not lead to useful pages. It can share a simplified version of this information with OpenAI’s crawler. The information described so far includes how fresh a page is, how good its traffic appears to be, and evidence that its content has changed.
What the Signals Cover
The signals help the crawler in two ways. Some point it toward pages that may be worth visiting and adding to the index. Others help it avoid wasting requests on things such as broken web addresses, websites that send almost every address to their homepage, and pages that repeat content already found elsewhere. Change signals also help the crawler decide whether it should visit a known page again or whether it would receive exactly the same content as before.
The companies are also testing signals that help identify unsafe pages. One example is a fake login form that an attacker adds to a single page on an otherwise trustworthy website. The aim is to flag that page so it can be left out of the search index without excluding the entire website. Early results suggest that these signals may help identify pages that should not be included in an index.
What the Signals Do Not Do
A signal that helps a crawler find a page does not give it permission to access that page. Website owners’ existing controls still apply. If a website blocks OpenAI’s search crawler, the page remains blocked even if Cloudflare’s network signals show that it is popular and has recently changed. Cloudflare says this clearly in its post, and Search Engine Journal identifies it as the program’s main limit.
The signals also do not decide where a page appears in search results. OpenAI finds and ranks the pages itself. Having a page added to the index does not guarantee that ChatGPT will mention it in an answer. Publishers would be wrong to assume that hundreds of millions of newly discovered pages mean hundreds of millions of new citations.
Why a 1% Group Can Be Small and Still Matter
One percent may sound small, but Cloudflare says its network operates in more than 335 cities and supports 45% of Fortune 500 companies. Even a group that makes up 1% of this network can include many different websites, from large retailers to small businesses. MACCUS described the result as Cloudflare helping ChatGPT find hundreds of millions of pages it had never seen before. This description gives a clearer idea of the scale than the percentage alone.
The small test group also raises another possibility. If a sample covering 1% of the network revealed so many unknown pages, the remaining 99% might contain many more pages that search engines have not found. This is an inference, not a reported result, and the companies have not made that claim. What the test does show is that finding pages through links leaves large gaps and that information about network traffic can help fill some of them.
Cloudflare has said it wants to make this ability available more widely. Search Engine Journal reports that earlier company posts suggested broader access might arrive during the rest of 2026. However, the October 15 announcement did not give a date. The next stage of the research will focus on choosing pages more effectively, making signals about page changes faster and more reliable, and measuring what happens after pages are discovered.
What the Companies Have Not Disclosed
What the announcement leaves out is just as important as what it reports. The post does not say how often the newly indexed pages appeared in ChatGPT answers. It does not say how many visitors reached those pages from ChatGPT. It also does not name the websites that took part in the test or explain whether the newly discovered pages belonged to those websites.
The post also provides no figures for how many pages were crawled or how much work the crawling placed on website servers. Reducing unnecessary server load for Cloudflare customers is one of the stated goals. However, the post admits that finding new pages may first increase crawling while the pages are checked and added to the index. Website owners might therefore see more requests from OpenAI’s crawler in their server logs before they see any extra visitors coming from ChatGPT.
Cloudflare says the companies are now studying whether better page discovery can lead to useful business results. Until they publish figures, the fair conclusion is that the pilot shows how many pages can be reached, not how much value those pages receive. Finding pages is the easier part of the problem. Whether people click through to visit those pages is what publishers care about.
Cloudflare Web Search API Pricing: Three Providers, a 28x Difference
Cloudflare’s Web Search API entered beta on October 2, 2026. It lets AI agents and other applications search the live web instead of guessing website addresses or relying only on information available before a model’s training cutoff. The service runs through AI Gateway, so each search appears in gateway logs and is charged against AI Gateway credits at the provider’s listed price. Cloudflare does not add its own markup. Developers can also use their own API key from a search provider.
Three providers were available at launch: Ceramic.ai, Exa, and Linkup. Cloudflare says all three support Zero Data Retention for requests made through its platform. This means they have agreed to the platform’s rules for not retaining request data. They have also agreed to Cloudflare’s standards for verified bots. These standards require crawlers to identify themselves, follow robots.txt rules, and include links to the original sources in their results.
| Provider | List price per 1,000 requests | Cost for 1 million requests | Multiple of cheapest |
|---|---|---|---|
| Ceramic.ai | $0.25 | $250 | 1x |
| Linkup | $5.00 | $5,000 | 20x |
| Exa | $7.00 | $7,000 | 28x |
| Cloudflare markup via AI Gateway credits | $0 | $0 | n/a |
| Bring-your-own provider key | Provider’s own rate | Provider’s own rate | n/a |
The prices for every 1,000 requests come from Cloudflare’s documentation dated October 2. The costs for one million requests are calculated by multiplying those prices. An AI agent that performs one million searches a month would pay $250 for the cheapest provider or $7,000 for the most expensive. This makes the provider choice a much bigger cost factor than other parts of the system. The higher prices may reflect better results or a larger search index, but the documentation does not provide benchmark tests comparing the providers. So the cheapest option is not automatically the worst choice.
Other AI companies are also building their own search services. DeepSeek launched its own search API alongside a new Flash model release. This means Cloudflare’s marketplace will compete with search tools offered directly by AI companies themselves.
How OpenAI’s Own Search Pricing Compares
OpenAI’s API pricing page, reportedly updated on October 23, lists web search at $10 per 1,000 calls. Search-content tokens are free, but normal charges for model tokens still apply. Our research found this price in only one source, so readers should treat it as a reported figure and check OpenAI’s pricing page before planning a budget. A separate OpenAI help-center price list for ChatGPT Enterprise describes the same $10 cost differently: it covers web searches, while search-content tokens are charged at the model’s usual rates.
If the $10 price is correct, OpenAI’s own tool costs 40 times as much as the cheapest provider available through Cloudflare and about 1.4 times as much as the most expensive one. However, this is not a perfect comparison. OpenAI’s tool combines finding information with its own models and ranking system. Cloudflare’s API returns search results that developers can use with whichever model they choose. A team already using OpenAI models may be willing to pay more for convenience. A team using several different models may prefer Cloudflare’s more flexible service.
OpenAI also says that API usage is billed separately from ChatGPT subscriptions, including Plus, Business, Enterprise, and Edu. Paying for one of these ChatGPT plans does not provide free API search credits. ChatGPT Search itself is available on Free, Go, Plus, Pro, Business, Enterprise, and Edu plans, subject to each plan’s usage limits. People can also use it without signing in.
Verified Bots Get Behavior Checks, Not Just Identity Checks
In a post published in mid-October, Chris Tsoukaladelis of Cloudflare explains that knowing a bot’s identity is not the same as knowing whether it can be trusted. A driver’s license shows who someone is and that they passed a driving test. It does not prove they will drive safely in the future. To earn Verified Bot status, a bot must identify itself correctly and securely, do what it claims to do, and avoid harmful behavior. Bots sign their requests using Web Bot Auth, and an open directory called BotBase keeps a record of them.
The new system checks the third requirement: behavior. Cloudflare now measures how often a crawler follows website owners’ preferences and uses these observations to create what it calls an explainable trust band. The band shows how consistently the bot behaves and which observed actions affected its assessment. Cloudflare stresses that this is not a permanent judgment about the bot or the company operating it.
The robots.txt and HTTP 429 Tests
The first check looks at whether a bot’s actual traffic follows the preferences a website publishes in its robots.txt file, in line with RFC 9309. Cloudflare says a single mistake may not be important. Instead, it looks for repeated behavior that ignores a website’s stated preferences. The company also points out that robots.txt sets minimum rules. If a file does not mention something, that does not automatically mean it is allowed.
The second check focuses on bots that visit permitted pages but send so many requests that they put too much pressure on a server. A website can respond with HTTP 429, meaning “Too Many Requests.” This tells the visitor to slow down and may include a suggested waiting period. Cloudflare checks whether the bot changes its behavior after receiving this response. Bots that regularly slow down or pause gain trust, while bots that keep sending requests at the same speed lose trust.
Cloudflare is already using the system internally to assess Verified Bots. It plans to publish the trust scores on Cloudflare Radar and eventually let all customers use the system. Customers would then be able to choose the trust level required to decide how Verified Bot traffic is handled. As of October 24, 2026, these public features have not launched. Cloudflare has also not published the exact scoring rules or any actual trust scores.
The Accountability System Cloudflare Is Building
Cloudflare describes its trust system as one part of a three-part process. Bot Preference Sync lets website owners publish their preferences and match them with their existing settings. The new behavior checks measure whether Verified Bots continue to follow those preferences over time. Bot Management then lets website owners decide how to handle each type of traffic.
These parts connect to Cloudflare’s search announcements. The Web Search API providers and the Verified Bots program follow the same basic rules: identify yourself, follow robots.txt, and slow down when asked. A search provider that wants to offer its service through Cloudflare must meet these requirements. A crawler that fails to follow them could lose its verified status, which can make access to Cloudflare-protected websites easier.
Cloudflare has taken action against bots before. Its post recalls that the company removed Perplexity’s bot from its Verified Bot list in 2025. According to the post, the bot tried to access resources on websites that had opted out of scraping and then tried to get around controls after being blocked. The new system aims to detect this kind of behavior sooner. It also aims to catch less serious problems caused by software bugs or incorrect settings rather than deliberate wrongdoing.
What Publishers Should Set in robots.txt Right Now
OpenAI operates separate crawlers for search and AI training, and the robots.txt rules for these crawlers are separate too. According to Search Engine Journal’s review of OpenAI’s crawler documentation, the available choices work as follows.
| robots.txt setting | Effect on ChatGPT Search | Effect on training use |
|---|---|---|
| Allow OAI-SearchBot, block GPTBot | Pages stay eligible for ChatGPT Search results | Signals that crawled content should not be used for training |
| Block OAI-SearchBot, allow GPTBot | Pages drop out of search results, though links may still appear as navigational links | Content may still be collected for training |
| Allow both | Pages stay eligible | OpenAI may use data from one crawl for both purposes |
| Block both | Pages drop out of search results | No training crawl |
The first option is the example OpenAI itself recommends in its documentation. Cloudflare customers can use AI Crawl Control, which is available on every plan, to see which AI services visit their websites and allow or block individual crawlers. OpenAI also says a website can qualify for ChatGPT Search by allowing OAI-SearchBot and allowing traffic from the search bot’s published IP addresses.
Developers who want to test the Web Search API in their own code can call it from a Worker using the AI binding or use the REST endpoint. Our guide to deploying a secure Cloudflare Workers API explains the basic steps for setting up the required token and routing.
Market Impact: Network Traffic Data Becomes a Product
For years, the network edge was mainly used to store copies of files closer to users and block attacks. This pilot treats the edge as a network of sensors that can collect information worth selling to search companies. If OpenAI’s results prove useful, other search engines and AI answer services may want similar deals. Internet infrastructure companies that already handle large amounts of traffic may also want to offer this service.
Cloudflare has a particular cost advantage. Information about which pages people visit comes from traffic its network already handles, so collecting the extra signals may cost relatively little. By combining these signals with bot controls, Cloudflare can serve both sides of the relationship. Publishers get more control over which crawlers visit their sites, while AI companies get a better way to find useful content. A company that only sells computing power may find this harder to copy. A similar advantage appeared when Cloudflare R2 competed with AWS S3 on data-transfer prices, showing how a network owner can offer lower costs for moving data than storage providers.
Cloudflare’s AI Search product adds a third part to this strategy. The product became generally available during Birthday Week. It now supports searches based on image pixels, text recognition in scanned PDFs, files up to 10 MiB, and any chat model. Usage-based billing begins on November 1, 2026. The service includes monthly allowances for adding information, storing it, and running semantic and full-text searches. Since the billing date is about a week away, teams already using AI Search should check their allowances before usage-based charges begin.
Publishers face a more difficult decision. Better discovery can bring more crawling, but it does not guarantee more visitors. The companies have not yet published referral figures. Websites that depend on appearing in search results may welcome the new signals. Websites that earn money from people visiting them directly may decide the trade-off is not worthwhile. It will be worth watching how the wider cloud computing market prices these network services when more customers can choose them.
Competitive Comparison: Where Cloudflare’s Edge Rivals Stand
Cloudflare competes with other edge computing and serverless providers. Our pricing comparisons show how this competition has developed. We compared Cloudflare Workers with AWS Lambda@Edge on speed and cost, and made a similar comparison between Workers and Fastly Compute. In both cases, the main question was which service could run code closest to users at the lowest price.
The search pilot moves the competition into a different area. We found no public announcement of a similar page-discovery deal between a major search company and any of Cloudflare’s main rivals. A large cloud provider can sell more computing power or serverless functions, but it cannot easily offer a view of what real visitors request across a large part of the web. Vercel, which we compared in our review of Workers and Vercel Edge, and Fastly both run edge networks. They could therefore pursue similar programs.
Cloudflare also has an advantage in bot control because of the large amount of traffic it can observe. An earlier Cloudflare post about the agentic web reported that its network handled more than 500 billion verified-bot requests per week at the time. A behavior score based on this amount of data may be more statistically useful than one based on a smaller network. However, a score only matters if bot operators care about it. That will depend on how many websites actually set trust thresholds and use them.
Historical Context: From Blocking Crawlers to Search Partnerships
Cloudflare’s relationship with AI crawlers has changed between strict enforcement and cooperation. Removing Perplexity from the Verified Bot list in 2025 was an example of enforcement. The OpenAI pilot is an example of cooperation. The new Verified Bots accountability system sits between the two approaches. It lets Cloudflare work with trusted bots while keeping the option to withdraw that trust if their behavior changes.
| Date | Event | Status |
|---|---|---|
| 2025 | Cloudflare de-lists Perplexity’s bot from its Verified Bot list | Completed |
| July 8, 2026 | Cloudflare announces the OpenAI discovery research | Completed |
| October 2, 2026 | Web Search API opens in beta through AI Gateway | Beta |
| October 15, 2026 | Cloudflare and OpenAI publish early pilot results | Research, ongoing |
| Mid-October 2026 | Verified Bots continuous accountability announced | Used internally |
| November 1, 2026 | AI Search usage-based billing begins | Scheduled |
| Rest of 2026 | Broader access to discovery signals, per Cloudflare’s stated intent | No date set |
The timeline shows how quickly these changes arrived. Within three weeks, Cloudflare opened a search marketplace, shared early results from the pilot, and changed how it checks whether bots behave well. The sequence looks like a planned rollout, although Cloudflare has not said that all these changes were designed to launch together.
What the Companies Say in Their Own Words
OpenAI described the pilot as an experiment to make search more efficient. Nick Ryder, identified in the coverage as OpenAI’s vice president of research, said according to Tech Times: “Up-to-date information is important for delivering accurate answers to people using ChatGPT. Piloting this with Cloudflare will allow us to explore whether network-level insights can help us discover content more efficiently.”
Cloudflare’s October 15 blog post describes the results more positively: “The pilot has brought previously unknown pages into OpenAI’s search index, giving ChatGPT more sources to draw on to help people find relevant, timely information.” The same post says that “Cloudflare’s network provides signals that can help OpenAI’s search crawler discover more useful pages across the web to surface to ChatGPT users.”
The post also explains who is responsible for ranking the pages. It says: “OpenAI then independently evaluates and ranks how those pages appear based on how relevant and helpful they are.” This is especially important for publishers. It makes clear that OpenAI decides how pages are ranked and, therefore, whether they receive traffic from search results.
Read together, the statements show that the companies are emphasizing different benefits. OpenAI talks about making search more efficient and answers more accurate. Cloudflare talks about reaching more pages and giving ChatGPT more sources. Neither company mentions revenue, traffic sent to publishers, or payments to website owners. These unanswered questions are likely to matter in future discussions.
Five Predictions for the Next Quarter
These points are predictions, not confirmed facts, and any of them could be wrong.
- Cloudflare expands the discovery program beyond the 1% test group. The company has said it wants to provide wider access during the rest of 2026. A test with only one partner is a limited starting point for a product. More customers may be able to opt in before the year ends, possibly alongside a second search company.
- Radar publishes the first Verified Bot trust bands. Cloudflare already uses the system internally and has said it plans to report results publicly. Publishing the information should be relatively easy and may encourage bot operators to improve their behavior. It is likely to happen before customers can set their own trust thresholds.
- Web Search API prices move closer together. A 28-fold price difference is difficult to maintain in a marketplace where customers can compare providers side by side. The cheapest provider may keep its price, while the most expensive may lower its price or offer features that justify the extra cost.
- Publishers use different robots.txt rules for search and training. Allowing search while blocking training is already OpenAI’s documented example, and tools such as AI Crawl Control make this setup easier. More websites may adopt it as they learn that appearing in search results and allowing training are separate choices.
- Other edge providers announce similar search-signal deals. The pilot gives competitors an example to follow. A provider with a large share of web traffic and a willing search partner could build a similar arrangement. At least one comparable announcement may follow.
FAQ: Cloudflare, OpenAI and ChatGPT Search
What did the Cloudflare OpenAI ChatGPT Search pilot achieve? The companies say it helped OpenAI discover and add hundreds of millions of previously unknown web pages to ChatGPT Search. The test used a group covering about 1% of Cloudflare’s network and began in July 2026. The companies have not shared an exact page count, how often the pages were cited, or how much referral traffic they received.
Does Cloudflare share my page content with OpenAI? The companies say the pilot uses web metadata to help with search and discovery, not shared page content for AI model training. A signal about a page does not give OpenAI permission to access it. Your robots.txt rules and Cloudflare controls still apply.
How much does the Cloudflare Web Search API cost? Cloudflare does not add a markup. When billed through AI Gateway credits, the service costs the provider’s listed price: $0.25 per 1,000 requests for Ceramic.ai, $5.00 for Linkup, and $7.00 for Exa. The service entered beta on October 2, 2026.
What is a Verified Bot trust band? It is an understandable rating range that shows how consistently a Verified Bot follows website preferences, including robots.txt rules and requests to slow down after an HTTP 429 response. Cloudflare says the rating is not a permanent judgment. Public reports on Cloudflare Radar and settings that customers can control are planned but are not yet available.
How do I keep my site in ChatGPT Search but out of AI training? Allow OAI-SearchBot and block GPTBot in your robots.txt file. OpenAI lists this as an example configuration. You should also allow traffic from OpenAI’s published search-bot IP addresses and use Cloudflare’s AI Crawl Control to check which crawlers visit your site.
Will finding more pages mean more visitors to my website? Not necessarily. OpenAI decides how pages are ranked, and the announcement includes no figures on referral traffic. The post warns that finding new pages may first increase crawling. You may therefore see more requests from crawlers in your server logs before you see more visitors.
Can all Cloudflare customers use the pilot? No. The project is still a research test involving a participating group. Cloudflare has said it wants to make the service more widely available, but it has not announced a launch date or product name.