Cloudflare and OpenAI: Making AI discovery better for everyone

Making your website easier for AI to find is not enough if AI cannot discover it in the first place. The answer to someone's question may already be on your website. It could be an installation guide that took you a week to write, or a first-hand review of a product with details that help someone decide whether to buy it. As more people use AI for help, you want AI systems to find your work, describe it correctly, and direct interested people to your website. You also want to control how they access your site.

Cloudflare and OpenAI launched a research pilot in July 2026 to explore these goals. They wanted to find out whether information from Cloudflare's network could help ChatGPT users discover new content more easily and quickly.

Today, we are sharing the early results. So far, the answer is a clear yes. The pilot has helped OpenAI add web pages it had not found before to its search index. This gives ChatGPT more sources to use when helping people find useful and up-to-date information. Along the way, we have learned more about what makes a page worth finding and how AI systems use the web.

Finding more of the web

The web contains many pages with specific details. Someone fixing a washing machine needs the manual for their exact model. Someone applying for a passport needs the latest rules and fees. Someone choosing a nearby restaurant needs its current menu and opening hours. Finding the right page can mean the difference between a wrong answer and a genuinely helpful one. These pages can be valuable even when very few other websites link to them.

In the past, making content easy to find mainly meant getting it near the top of a search results page. After that, the person visiting the page decided whether the information was useful and could take action, such as booking a table or applying for a passport. As more people use AI to find information and complete tasks, discoverability has taken on a new meaning. AI systems need to find the right sources, understand which ones are useful, and connect people with relevant information. They can do this by giving an answer, citing a source, or sending someone to the website itself.

Cloudflare's network provides information that can help OpenAI's search crawler find more useful pages across the web and show them to ChatGPT users. OpenAI then independently checks these pages and decides how to rank them based on how relevant and helpful they are. Working together in this way can help both systems find more content and improve the quality of the information people discover.

This joint pilot used a group that made up about 1% of Cloudflare's entire network. It has helped OpenAI find and add hundreds of millions of new web pages to its search index. This helps people discover more useful content and get more questions answered through ChatGPT. ChatGPT Search also links to relevant sources, so people can read the original content and find the websites and publishers behind it.

With GPT-6 and Intelligent UI, OpenAI is also making ChatGPT a better place to shop, learn, and find answers. It offers more visual and interactive experiences that are designed around what people need. This project supports those improvements by helping ChatGPT find more useful, up-to-date information across the web.

From discovery to useful answers

Finding a web address is only the first step. We also need to know whether people can open the page, whether it contains something different or useful, and what it could help someone do.

For example, we have learned that automated traffic can make pages that do not exist look interesting. Bots may repeatedly ask for files that are missing. Some websites send almost every web address to their homepage. Other addresses lead to copies of the same content. Recognizing these patterns lets us avoid wasting time before a crawler downloads a page. This makes the process more efficient and can also benefit website owners.

We are also testing signals that can identify and flag pages that may be harmful. This adds another layer of information to help make discovery safer and more reliable. Even a trustworthy website can have one page that has been taken over by an attacker, such as a fake login form. Flagging that single page instead of the entire website helps protect people without unfairly penalizing the site owner.

Early results suggest that these signals can help identify pages that should not be included in a search index.

Making discovery easier

As more people use AI to find information, website owners have new ways to reach people who are interested in their content. For owners who choose to make their content available to AI systems, these improvements make it easier and more efficient for those systems to find it.

Downloading every page more often to keep information up to date would cost everyone more time and resources. For website owners, a crawler that repeatedly downloads a page that has not changed creates extra work for the website without providing any benefit.

The signals described above help prevent this waste. Filters keep crawlers away from pages that do not exist or simply send visitors somewhere else. Signals about changes tell crawlers when a page is worth visiting again and when it is not.

Getting the balance right is important. Finding new pages may initially lead to more crawling while those pages are checked and added to the search index. The goal is to keep finding new content and make sure information stays current while reducing unnecessary work. For Cloudflare customers, this means less extra traffic on their websites. For OpenAI, it means more search resources can be used to find information that helps people.

Making it worthwhile to participate

The next stage will build on what we have learned. We plan to improve how pages are selected, test faster and more reliable ways to detect changes, and measure what happens after content is discovered.

We are continuing to work with OpenAI to understand how better website discovery can lead to real business results. We will keep sharing what we learn as the work continues.

Website owners' choices remain important throughout this process. A signal about a page does not give permission to access it. This research uses information about web pages to help with search and discovery. It does not share page content for training AI models.