AI Citation Starts With Crawler Access, Not Schema

AI assistants can only cite what their crawlers are allowed to fetch, so GEO readiness starts as a permissions problem, not a content problem. Before you rewrite a page for AI search, check which AI crawlers your robots.txt actually allows, because a blocked GPTBot, ClaudeBot, or PerplexityBot makes every other optimization irrelevant.

Most GEO checklists jump straight to schema markup and llms.txt files, treating both as the technical foundation of AI-search visibility. The 2026 data tells a less tidy story. Crawler access is the real gate, some of the tactics marketers lean on hardest barely move the needle, and the two problems get treated as one when they should be handled in the opposite order.

Can AI Assistants Actually Reach Your Site?

Not automatically, and not uniformly across AI providers. Each AI company runs more than one crawler, and each one does a different job, so a single robots.txt line rarely covers what a marketer thinks it covers.

OpenAI's own crawler documentation draws a clear line between GPTBot, which gathers content for model training, OAI-SearchBot, which indexes pages for ChatGPT's search feature, and ChatGPT-User, which fetches a page live when someone asks ChatGPT about it mid-conversation. A site can disallow GPTBot to opt out of training while still allowing OAI-SearchBot and ChatGPT-User, and stay fully citable in ChatGPT answers. Flip that logic, block the wrong one, and you can keep your training opt-out while accidentally cutting yourself out of citations entirely. The same three-way split exists for other providers under different names.

Get these tokens confused in a client's robots.txt and you get an outcome nobody intended: content that still shows up in classic organic search but never surfaces in an AI answer, with no error message telling anyone why.

Why Did Cloudflare Just Flip Everyone's Defaults?

Because most site owners were leaving the choice unmade, and an unmade choice on a network Cloudflare controls is still a choice. On its own blog, Cloudflare announced in July 2026 that it was sorting AI traffic into three declared categories, Search, Agent, and Training, and that as of September 15, 2026, new domains and existing free-tier sites that run ads would have Training and Agent traffic blocked by default while Search stayed allowed.

That single infrastructure decision reset the baseline for a huge share of the small-business web, including plenty of MSP client sites that sit on Cloudflare's free tier without anyone on the marketing side ever opening the dashboard. A client can rank well in Google, run a normal ad-supported blog, and still have quietly gone dark to ChatGPT's ChatGPT-User fetches and Claude's live retrieval the moment that default kicked in, simply because nobody checked the traffic-category settings after the change shipped.

How Widespread Is AI Crawler Blocking Right Now?

Wide enough that it is now a defined market segment rather than a fringe choice. Independent tracking from Presenc AI's robots.txt adoption research finds that GPTBot blocking climbed from around 5 percent of top sites in early 2023 to roughly 25 percent by 2024, and has held near that level through 2026, with ClaudeBot now the fastest-growing bot being added to disallow lists. The pattern that shows up most often is not a blanket block. It is a split decision, training crawlers disallowed, search and answer crawlers allowed, made by site owners who want AI visibility without donating training data.

That split matters for MSPs because it means blocking is not evidence of a technically unsophisticated client. Plenty of these sites made a deliberate, defensible choice. The problem shows up when a client blocked everything with one copy-pasted robots.txt snippet years ago and never revisited it once citation traffic became a channel worth protecting.

Does an llms.txt File Actually Get You Cited?

Not on the evidence so far. Ahrefs analyzed 137,000 domains and found that among the roughly 38,000 with a published llms.txt file, 97 percent received zero requests for it in the month studied. Of the small share that did get hit, the majority of requests came from SEO audit tools, uncategorized bots, and site-profiling crawlers checking whether the file existed, not from GPTBot, ClaudeBot, or other model crawlers actually reading it. Ahrefs found no case of an AI crawler requesting an llms.txt file that did not already exist, meaning the bots are not out looking for one.

That does not make llms.txt worthless as documentation for your own team, but it means treating it as a citation lever is the wrong framing. It is a manifest, not a magnet. If a client's technical SEO plan lists "add llms.txt" as a headline GEO tactic, the honest correction is that it belongs several rungs below fixing crawler access and page-level content quality, not above them.

Does Adding Schema Markup Move AI Citations?

Barely, according to the most rigorous test run on it. Ahrefs tracked 1,885 pages that added JSON-LD schema between August 2025 and March 2026, matched them against roughly 4,000 similar pages that made no schema change, and measured citation movement across Google AI Overviews, Google AI Mode, and ChatGPT. AI Overview citations moved -4.6 percent relative to the control group, a small decline, while AI Mode and ChatGPT citations moved +2.4 percent and +2.2 percent, both too small to separate from noise. None of the three results support schema markup as a reliable citation lever on its own.

Google has been narrowing this space too. Its own FAQPage structured data documentation now carries a deprecation notice stating that FAQ rich results stopped appearing in Google Search as of May 7, 2026, with related Search Console reporting for FAQ markup being removed over the following months. The schema type still validates and still describes your content correctly to any system parsing it, but the visual reward that made marketers chase it in the first place is gone, and the AI-citation reward was never well established to begin with.

None of this means schema is pointless. It means it functions as a clarity aid for whatever content already earns attention, not as a substitute for that content, and definitely not as a stand-in for making sure the crawler can reach the page at all.

Building the GEO Access Checklist Clients Actually Need

Put crawler access ahead of markup on every technical GEO engagement, in this order:

• Pull the live robots.txt for the domain and check GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, PerplexityBot, and Google-Extended by name, not by a wildcard AI disallow rule that may be blocking more than intended.
• Confirm the site's CDN or security platform settings, not just the robots.txt file, since Cloudflare and similar providers can now enforce traffic categories independent of what the text file says.
• Treat schema and llms.txt as hygiene, done once content and access are sound, not as the first two line items on the GEO scope of work.

This is exactly the kind of layered technical check that is hard to deliver consistently by hand across a client book, and it is why a standardized scan matters more than a one-off audit. Running that check through ActiScore means you catch a misconfigured crawler directive on the same pass that flags a schema or content issue, instead of finding out three months later that a client went uncited the whole time. If you want to turn that audit into a recurring, branded deliverable rather than a spreadsheet you rebuild every quarter, Stack Builder packages it as a service you can sell under your own name.

What This Means for Your Own Service Line

GEO access audits are a natural add-on next to the security and compliance scans MSPs already run, because the underlying skill, reading a client's technical configuration and translating it into a fixable action item, is the same one. You are not learning a new discipline. You are pointing an existing one at a newer set of crawlers.

Selling it also solves a retention problem. A client who understands that their AI visibility depends on settings inside their own CDN dashboard, not just their content calendar, has a concrete reason to keep a technical partner on retainer instead of treating marketing as a project that ends. That reframes GEO from a content service into an infrastructure service, which is a stickier position for an MSP to hold.

Actiforge builds the tooling so you do not have to build a crawler-monitoring script from scratch or track every vendor's bot-naming changes yourself. Browse the rest of the white-labeled toolset built for exactly this kind of technical service expansion. See the full stack.