ChatGPT Search Can Draw On More Pages After A Cloudflare Pilot

Cloudflare and OpenAI say a research test helped OpenAI find and add hundreds of millions of new web pages to its search index. The test used a group of websites that made up about 1% of Cloudflare’s entire network.

The companies shared these early results on October 15 in a post on Cloudflare’s blog written together with two OpenAI staff members. Cloudflare announced the test on July 8.

The post says that finding new pages can cause crawlers to visit websites more often at first. It does not say how often ChatGPT cited the new pages or how many visits those pages brought to websites.

How Cloudflare’s Information Helps OpenAI Find Pages

OpenAI uses information from Cloudflare’s network to help its search crawler find useful web pages. The blog post gives several examples: a user guide for a particular washing machine, the latest details about passport rules and fees, and a restaurant’s menu and opening hours. The post says these pages “can be valuable even if few other sites link to them.”

Cloudflare’s July 8 announcement described some of the new information the system would use. This included how recently a page had changed, the quality of traffic to a website, and what had actually changed on a page. The announcement also says this information comes from websites that take part in the project. However, neither announcement names the participants or says whether the new pages are on their websites.

How the Information Changes Crawling

The October 15 post says Cloudflare’s information will also help OpenAI’s crawlers avoid unnecessary requests. Bots sometimes keep trying to download files that a server does not have. Some websites send almost any URL to their homepages, while other URLs lead to copies of the same content. Information about changes helps the crawler decide whether it should visit a page again or whether another visit would not be useful.

The companies are also testing information that can flag pages that might be harmful. For example, an attacker might add a fake login form to one page on a legitimate website. The aim is to flag that page, rather than treat the whole website as harmful. Early results suggest this information can help identify pages that should not be added to a search index, the post says.

The post says the goal is to reduce the extra work placed on Cloudflare customers’ servers. However, it gives no numbers showing how many crawler requests were made or how much server load was reduced. It also says, “Finding new pages can initially increase crawling as those pages are evaluated and indexed.”

Website Crawler Settings Still Control Access

The October 15 blog post states, “A signal about a page doesn’t grant permission to access it.” It also says the researchers use information about web pages to improve how search engines find and discover content, rather than share the page content for training AI models.

The document explaining how OpenAI’s crawler works uses two separate robots.txt rules for search and training. If a website blocks OAI-SearchBot, the bot that searches for information, that website will not appear in ChatGPT’s search results. However, the website might still appear as a link that users can follow to reach a specific destination. If a website allows both OAI-SearchBot and GPTBot, the bot that collects content that may be used for training, OpenAI may use data collected during one crawl for both purposes.

The October 15 post points readers to Cloudflare’s AI Crawl Control, which is available on every plan. This service shows which AI services visit a website and lets website owners allow or block each crawler.

Why This Matters

Once a page is added to OpenAI’s index, ChatGPT has another source it can use to find information. The companies say OpenAI checks and ranks indexed pages independently. This means that finding a page through the test does not guarantee that ChatGPT will cite it.

If a website allows OAI-SearchBot but blocks GPTBot, ChatGPT can still show the website in search results, while the settings tell OpenAI not to use the content collected by GPTBot for training. OpenAI gives this same setup as an example on its crawler information page.

Website owners can track visits from ChatGPT using a UTM tag or a referrer, which can show where a visitor came from. However, some ChatGPT links had neither earlier this month, as I reported on October 15. One analyst said a fix started rolling out that day.

What Happens Next

Cloudflare and OpenAI say the next stage of the project will focus on helping AI systems choose web pages more effectively, testing faster and more reliable ways to detect changes to content, and measuring what happens when the AI finds new content.

Cloudflare says it is working with OpenAI to find out how better content discovery could lead to business benefits.

In a July 1 blog post about its research into network information and AI search, Cloudflare said it planned to make this capability widely available sometime during the rest of 2026. I also mentioned this plan in my September report. The October 15 announcement did not give a specific release date.

For now, website owners can see OpenAI’s crawler activity in their server logs and in AI Crawl Control.

Featured Image: Tetiana Yurchenko/Shutterstock