Cloudflare and OpenAI: Making AI discovery better for everyone

Making your website easy for AI to understand is not very useful if AI cannot find it. The answer to someone's question may already be on your website. It could be an installation guide that took you a week to write or a first-hand account of using a product. These details may help someone decide whether to buy it. As more people turn to AI for help, you want AI systems to find your work, describe it correctly, and direct interested people to your website. You also want to control how these systems access your site.

Cloudflare and OpenAI launched a research pilot in July 2026 to explore these goals. They wanted to find out whether information from Cloudflare's network could help ChatGPT users find new content more easily and quickly.

Today, we are sharing the early results. So far, the results are very positive. The pilot has helped OpenAI find web pages that were not previously known to its search system and add them to its search index. This gives ChatGPT more sources to use when helping people find relevant, up-to-date information. During this work, we have also learned more about what makes a page worth finding and how AI systems use the web.

Finding more of the web

The web contains many useful details. Someone fixing a washing machine needs the manual for their exact model. Someone applying for a passport needs the latest rules and fees. Someone choosing a local restaurant needs its current menu and opening hours. Finding the right page can make the difference between a wrong answer and one that truly helps. These pages can be valuable even if few other websites link to them.

In the past, making a website easy to find mostly meant getting it near the top of a search results page. After that, the person visiting the site decided whether the information was useful, booked a table, or applied for a passport. As more people rely on AI to find information and complete tasks, being easy to find has taken on a new meaning. AI systems need to find the right sources, understand which information is useful, and connect people with it. They can do this through an answer, a citation, or a visit to the original website.

Cloudflare's network provides information that can help OpenAI's search crawler find more useful pages across the web and show them to ChatGPT users. OpenAI then independently decides how relevant and helpful those pages are and ranks them. This process can improve over time, helping both systems find useful pages and helping people get better information.

This joint pilot used a participating group that made up about 1% of Cloudflare's entire network. It has helped OpenAI find and add hundreds of millions of new web pages to its search index. This can help people discover more useful content and get answers to more questions through ChatGPT. ChatGPT Search also links to relevant sources, so people can explore the original content and find the websites and publishers behind it.

With GPT-6 and Intelligent UI, OpenAI is also making ChatGPT a better place to shop, learn, and find answers. It offers more visual and interactive experiences designed around what people need. This work supports these improvements by helping ChatGPT find more useful and up-to-date information from across the web.

From discovery to useful answers

Finding a web address is only the first step. We also need to know whether people can open the page, whether it offers something useful or different, and what it might help someone do.

For example, we have learned that automated traffic can make a page that does not exist seem worth visiting. Bots may repeatedly request files that are not there. Some websites send almost any web address to their homepage. Other addresses lead to copies of the same content. Recognizing these patterns helps us avoid wasting time before a crawler downloads a page. This makes the process more efficient and can benefit website owners.

We are also testing ways to identify and flag pages that may be harmful. These signals provide another layer of information to make finding web content safer and more reliable. A real website may have just one page that an attacker has changed, such as a fake login form. Flagging that page instead of the entire website helps protect people without unfairly penalizing the site owner.

Early results suggest that these signals can help identify pages that should not be included in a search index.

Making discovery easier

As more people use AI to find information, website owners have new ways to reach people interested in their content. For website owners who choose to make their content available to AI systems, these improvements can make it easier and faster for those systems to find it.

It would cost everyone more if crawlers had to check every web page more often. For a website owner, a crawler that repeatedly downloads a page that has not changed creates extra work without providing any benefit.

The signals described above help prevent this waste. Filters keep crawlers away from pages that do not exist or that send visitors somewhere else. Signals about changes tell crawlers when a page is worth checking again and when another visit is not needed.

Getting the balance right is important. Finding new pages may initially increase crawling while those pages are checked and added to search indexes. The goal is to keep finding new pages and keeping information up to date while reducing unnecessary work. For Cloudflare customers, this means less unnecessary traffic on their websites. For OpenAI, it means more of its search resources can be spent finding information that helps people.

Making it worthwhile to participate

In the next phase, we will build on what we have learned. We will improve how pages are selected, test faster and more reliable ways to detect changes, and measure what happens after content is found.

We are continuing to work with OpenAI to understand how better website discovery can lead to real business results. We will keep sharing what we learn as the work continues.

Website owners' choices remain important throughout this process. A signal about a page does not give anyone permission to access it. This research uses information about web pages to help with search and discovery. It does not share page content for training AI models.

You can use Cloudflare's AI Crawl Control to understand and manage how AI systems access your website.

Disclaimer: Pitchfork Partners provided the press release above. CXO Digital Pulse is not responsible for its content in any way.
Copying or reproducing any part or all of the content is not allowed without the author's approval.