Your sitemap is a wishlist. Your bot logs are the truth

Most teams think their sitemap shows what's on their site. It doesn't. A sitemap is a wishlist: the pages you want crawlers to find. The pages bots actually visit are a different list, and that list lives in one place only, your CDN logs.

The gap between those two lists is where the problems hide. So we ran the comparison on our own site, Limy. The method is simple: take every page in the sitemap, take every page bots actually visited in the CDN logs, line them up, and mark each one as in the sitemap, in the logs, or both.

Here's what we found.

Two of the three groups are good news. The third is the whole story.

This used to be a niche SEO task. Not anymore, because who reads your pages has changed. As of mid-2026, Cloudflare Radar says about 35% of all web requests are bots, roughly one in three. AI crawlers alone are about 20% of verified bot traffic, and AI-search bots add more on top. The web grew 19% in 2025, and bot traffic grew faster than human traffic.

The growth is the part to watch. HUMAN Security found AI web traffic grew 187% in 2025, and AI agents, bots acting for a person, grew thousands of percent year over year. Cloudflare saw GPTBot's requests jump 305% between May 2024 and May 2025. These aren't the quiet crawlers of five years ago feeding a search page nobody scrolls. They feed what ChatGPT, Claude, Perplexity, and Gemini say about your brand.

When most visits are machines, which pages they read really matters.

When this group is empty, every page you listed is being found. All 110 of our listed pages showed up in real bot traffic, even the messy ones, like a blog link ending in a period and a customer-story link with an odd double-encoded address. No forgotten announcements, no product pages the bots skip, no launch content left out in the cold.

Most teams assume this check passes and never test it. It fails more often than you'd think, usually because the menu is built with JavaScript. Most AI crawlers don't run JavaScript (more on that below), so a page that's only linked by script has no way in. A site that looks fully linked to Google can look like scattered islands to GPTBot. When that happens, pages quietly drop out: you publish, you add it to the sitemap, and months later nobody notices the bots never came.

For AI search, a page that's never visited can't be cited, mentioned, or pulled into an answer. It's invisible, no matter how good it is. An empty sitemap-only group is proof that every page you meant to publish can actually be found.

This is the real finding, and almost nobody looks here. The sitemap is the list of pages you want visited. The logs show where bots actually spend their time. When the log-only list is seven times bigger than the one you chose, bots are spending most of their effort on pages you never picked.

That matters because a bot only visits so many pages, and AI crawlers burn through that budget while sending almost no one back. Cloudflare Radar data from early 2026 shows some AI bots fetch tens of thousands of pages for every single visitor they send you.

Every one of those fetches hits a real page on your site. When 814 of the pages bots visit are ones you never listed, a big chunk of all that crawling is landing on content that doesn't speak for your brand.

On our site, those 814 pages fall into a few familiar groups.

This isn't just us; it's how most of the web looks. Vercel and MERJ studied over 500M AI-crawler fetches and found these bots hit dead pages about one in three times, over 34% ending in a 404, versus about 8% for Googlebot. Most of that waste is exactly what the CDN-only group shows: old sitemap entries, broken links, and renamed pages the bot still thinks are live. AI crawlers keep a list of these dead pages and keep visiting them. Every wasted visit is a real, live page somewhere else on your site the bot didn't reach.

Old search engines were forgiving. They'd index the junk, rank it low, and move on. AI doesn't rank, it reads everything and writes an answer. When it answers a question about your company, it pulls from whatever it read, and you don't control the mix. A stale help page or a half-built draft doesn't just sit at the bottom where no one looks. It becomes part of what AI says about you.

That raises the stakes of the CDN-only group. Every stray page in it can quietly shape how AI describes you, with none of the care you'd give a page you chose to publish. The sitemap is what you meant to say. The logs are what AI is actually reading. The gap between them is your risk.

This has to be done with CDN logs, not your analytics tool, because your analytics can't see these bots at all. Analytics runs on JavaScript, and the big AI crawlers don't run JavaScript. Vercel and MERJ checked more than 500M GPTBot fetches and saw no JavaScript running, and the same was true for ClaudeBot and PerplexityBot. ClaudeBot even downloaded JavaScript files about a quarter of the time and never ran them. The one real exception is Google's Gemini crawler, which uses Google's own rendering.

The result is blunt: a page built with JavaScript can rank fine on Google but look empty to ChatGPT, Claude, and Perplexity, and any tracking that needs that same JavaScript never fires for those bots. Your analytics shows you humans. It doesn't show you the machines that are now most of your traffic. CDN logs are, as one 2026 analysis put it plainly, the only reliable source, and comparing AI bot visits against your sitemap is what shows the gap.

The point isn't our 814 pages. It's the comparison itself, and any brand can run it.

Then read the three groups in order. Sitemap only is your "can't be found" list: fix the links, or accept it's invisible to AI. CDN only is your risk list: go page by page and decide if you want bots there, then clean up the rest. Both is the healthy core.

It only works with CDN logs. You can't compare against traffic you can't see, and normal analytics never sees these bots because they don't run JavaScript. The logs are the only place the truth lives. The sitemap says what you published. The logs say what the machines actually read.