A quiet infrastructure experiment between Cloudflare and OpenAI just changed what "being visible to AI search" means. On October 15, the two companies shared early results of a research pilot, coauthored on Cloudflare's blog with two OpenAI staff: signals from a participating cohort of roughly 1 percent of Cloudflare's network helped OpenAI discover and index hundreds of millions of new web pages — many of which no search engine had previously bothered to crawl. Search Engine Journal covered the announcement on October 20.
That matters because of which pages got found: a washing machine's user guide, current passport requirements and fees, a restaurant's menu and hours. "Small" pages with real answers — the exact kind of content small businesses publish and that classic SEO has always told them is invisible without backlinks. This piece explains how the pilot works, what it does not change, and what to do about it this month.
What the pilot actually is
Cloudflare announced the research experiment on July 8, 2026. The idea: OpenAI's search crawler traditionally discovers the web the way search engines always have — by following links. But link graphs are a historical map, not a live one. Pages that few sites link to, pages deep in a site, pages that changed since the last crawl — the link-following approach misses them or arrives late.
The pilot feeds OpenAI a different discovery layer built from Cloudflare's network signals volunteered by participating customers:
| Signal type | What it tells the crawler | Why it matters |
|---|---|---|
| Recency of updates | When a page actually changed | Fresh answers get crawled sooner, stale ones skipped |
| Traffic quality | That real humans find the page valuable | A relevance signal independent of links |
| What changed | Whether the edit was substantive or cosmetic | Prioritizes meaningful content changes |
| Dead-end flags | Broken URLs, soft redirects, duplicate copies | Crawl budget stops being wasted |
| Malicious-page flags | Example given: a fake login form planted on one page of a legitimate site | Bad pages excluded page-by-page, not domain-wide |
The October 15 update reported the early outcome — hundreds of millions of newly discovered and indexed pages from about 1 percent of Cloudflare's network — along with two honest caveats: the companies did not report how often the new pages are actually cited in ChatGPT answers or how many visits they send, and finding new pages can initially increase crawling as they are evaluated.
Discovery is not permission
The single most important sentence in Cloudflare's post: "A signal about a page doesn't grant permission to access it."
Discovery and access are now officially separate layers. OpenAI still divides its crawlers by purpose, with separate robots.txt directives for search versus potential model training. The practical configuration, which OpenAI itself suggests as an example, is allowing OAI-SearchBot (search indexing) while blocking GPTBot (training data collection): your content stays visible in ChatGPT search results without being used to train models. Block OAI-SearchBot instead and your site stays out of ChatGPT's search index, though it may still appear as a navigational link. Cloudflare's AI Crawl Control, available on every plan, is where you can see which AI services are actually hitting your site and allow or block each one — and your own server logs show the same crawler activity directly.
Why this is a big deal for small sites
For two decades, the unwritten rule of search was: no links, no visibility. The pages that made it into indexes were the ones the link graph blessed. That bias baked itself into content strategy everywhere — including the advice most small businesses received, which was to obsess over backlinks instead of answering questions well.
The Cloudflare pilot is early evidence that AI search discovery is being rebuilt on a different foundation: direct behavioral and freshness signals from the network layer. Cloudflare's examples make the point explicitly — the pilot's showcase pages are "valuable even if few other sites link to them." A restaurant's hours, a product manual, an agency's fee schedule: pages whose authority comes from being genuinely useful when a person needs them, not from citation count.
That reframes the advice we have been giving since our three-doors audit of AI visibility: the door that matters most is being machine-readable, current and genuinely answer-worthy at the page level. Now there is public, named infrastructure from two of the biggest players on the internet moving discovery in exactly that direction.
Duotone ribbons through the fog: what still is not solved
Temper the optimism. Three big unknowns remain:
- No performance data. Newly indexed does not mean newly cited. OpenAI says pages found through the pilot are ranked on their own merits — being discoverable gets you a ticket, not a stage.
- Attribution is still messy. As we documented during October's ChatGPT attribution bug, visits from ChatGPT can lose their tags; measure with more than one signal and keep server logs.
- Access is uneven. A pilot drawn from 1 percent of Cloudflare's network helps sites on Cloudflare that opted in; broad access is planned "at some point during the remainder of 2026," with no date announced. Most sites today are still discovered the old way.
What to do this month
The pilot strengthens the case for a boring, high-leverage checklist:
- Decide your crawler posture deliberately. Allow OAI-SearchBot unless you have a reason not to; decide about GPTBot on training-policy grounds, not by accident. Check AI Crawl Control or your logs to see who is already visiting.
- Kill the crawl dead ends. Broken URLs, mass redirects to the homepage and duplicate copies are now measurable waste — the pilot explicitly trains crawlers to skip them.
- Make your deep pages worth finding. Guides, menus, hours, fee pages, specifications — concrete, current, well-structured. Write the individual page, not just the category page; that is where this discovery layer looks.
- Signal change honestly. Freshness signals only help if updates are real. Update the page's content, and keep metadata truthful — no fake-churn tricks.
- Watch Bing too. Microsoft is investing in similar AI-grounding infrastructure; the same hygiene pays off across engines. Bing Webmaster Tools' Topics Portfolio, which we covered earlier this month, shows some of the same shift from link-based to topic-based evaluation.
Conclusion
The Cloudflare-OpenAI pilot is the clearest public signal yet that AI search discovery is being rebuilt on network-layer evidence — recency, traffic quality and genuine change — rather than the link graph that small sites could never win. Early results: a 1 percent network cohort handed OpenAI hundreds of millions of newly indexed pages, including exactly the kind of unglamorous, high-answer content small businesses own. The caveats are real (no citation or traffic numbers yet, uneven access, rankings still earned independently), but the direction favors businesses whose pages are honest, current and genuinely useful. Get your crawler posture deliberate, your dead ends cleaned and your deep pages worth finding — that work compounds no matter which discovery engine wins. Want help auditing exactly where you stand with AI search? Contact us; auditing machine-readability is one of our core services.
