Cloudflare and OpenAI: Making AI discovery better for everyone

Making your website easier for AI to understand is not much help if AI cannot find it.

The answer to someone's question may already be on your website. It might be an installation guide that took you a week to write, or a personal account of using a product that includes details that help people decide whether to buy it. As more people use AI for help, you want these systems to find your work, describe it correctly, and direct interested people to your website. You also want to control how they access your site.

Cloudflare and OpenAI started a research trial in July 2026 to explore these goals. They wanted to find out whether information from Cloudflare's network could help ChatGPT users find new content more easily.

Today, we're sharing the early results. So far, the results are very positive. The trial has helped OpenAI's search system find web pages it did not know about before. This gives ChatGPT more sources to use when helping people find useful, up-to-date information. Along the way, we've learned more about what makes a page worth finding and how AI systems use the web.

Finding more of the web

The web contains lots of specific information. Someone fixing a washing machine needs the manual for their exact model. Someone applying for a passport needs the latest rules and fees. Someone choosing a local restaurant needs its current menu and opening hours. Finding the right page can mean the difference between a wrong answer and one that truly helps. These pages can be useful even if very few other websites link to them.

In the past, being easy to find on the web mostly meant appearing near the top of a search results page. After visiting a website, people had to decide whether its information was useful, book a table, or apply for a passport themselves. Now that more people rely on AI to find information and complete tasks, being easy to find means something more. AI systems need to locate the right sources, understand what is useful, and connect people with relevant information. They can do this through an answer, a citation, or a visit to the website itself.

Cloudflare's network provides information that can help OpenAI's search crawler find more useful pages across the web and show them to ChatGPT users. OpenAI then independently checks and ranks those pages according to how relevant and helpful they are. Together, these steps create a cycle that can improve both how pages are found and the quality of information people discover.

This joint trial used a group that represented about 1% of Cloudflare's entire network. It has helped OpenAI discover and add hundreds of millions of new web pages to its search index. This helps people find more useful content and get more questions answered through ChatGPT. ChatGPT Search also links to relevant sources, allowing people to explore the original content and find the websites and publishers that created it.

With GPT-6 and Intelligent UI, OpenAI is also making ChatGPT a better place to shop, learn, and find answers. It offers more visual and interactive experiences that adapt to what people need. This work supports those improvements by helping ChatGPT find more useful, up-to-date information across the web.

From discovery to useful answers

Finding a web address is only the first step. We also need to know whether people can open the page, whether it offers something different from other pages, and how it might help them.

For example, we've learned that automated traffic can make a page that does not exist look interesting. Bots may repeatedly request files that are not there. Some websites send almost every web address to their homepage. Other addresses lead to copies of the same content. Recognizing these patterns lets us avoid wasted work before a crawler downloads a page. This makes the process more efficient and benefits website owners.

We're also testing signals that can identify and flag pages that may be harmful. These signals provide another layer of information to help make discovery safer and more reliable. A real, trustworthy website might have one hacked page, such as a fake login form added by an attacker. Flagging that specific page instead of the entire website helps protect people without unfairly penalizing the site owner. Early results suggest these signals can help identify pages that should not be included in a search index.

Making AI Discovery Easier

As more people use AI to find information, website owners have new ways to reach people who are interested in their content. For website owners who choose to make their content available to AI systems, these improvements make it easier and more efficient for those systems to find it.

Trying to keep up by downloading every page more often would cost everyone more. For website owners, a crawler that repeatedly downloads a page that has not changed puts extra demand on their website without providing any benefit.

The signals described above help prevent this waste. Filtering stops crawlers from spending time on pages that do not exist or simply send visitors somewhere else. Signals about changes tell crawlers when a page is worth visiting again and when it is not.

Getting the balance right is important. Finding new pages can initially lead to more crawling while those pages are checked and added to the search index. The goal is to keep finding new pages and updating information while reducing the work needed to do so. For Cloudflare customers, this means less unnecessary demand on their websites. For OpenAI, it means more search resources can go toward information that helps people.

Making it worthwhile to participate

The next stage will build on what we've learned. We'll improve how pages are selected, test faster and more reliable ways to detect changes, and measure what happens after content is discovered.

We're continuing to work with OpenAI to understand how making websites easier to find can lead to real business benefits. We'll keep sharing what we learn as the work continues.

Website owners' choices remain important throughout this process. A signal about a page does not give anyone permission to access it. This research uses information about the web to help with search and discovery. It does not share page content for training AI models.

You can use Cloudflare's AI Crawl Control to understand and manage how AI systems access your website.