# AI discoverability — crawlers, rendering, llms.txt

How AI assistants (ChatGPT, Claude, Perplexity, Gemini, Copilot) find and cite
a site — and the two silent failure modes that make a "perfectly SEO'd" site
invisible to them anyway. Crawler/rendering tables adapted from
[claude-blog](https://github.com/AgriciDaniel/claude-blog) (MIT © AgriciDaniel).

## Google needs nothing special — say so when asked

Google's official guidance
(https://developers.google.com/search/docs/fundamentals/ai-optimization-guide):
AI Overviews and AI Mode run on Google's normal Search ranking systems. No
special markup, no AI schema, no llms.txt — **Google Search ignores llms.txt
entirely**. Normal SEO (the main checklist) *is* the Google-AI strategy. If
the user asks about "GEO"/"AEO" tactics or paid AI-optimization services,
this is the honest answer: for Google it's just SEO; for other assistants
the extras below help.

llms.txt still earns its place for **non-Google** assistants — keep writing
it, but present it as that, never as a Google requirement.

## Three kinds of AI bot — three different decisions

| Kind | Bots | Blocking means |
|---|---|---|
| Search-indexing | OAI-SearchBot, Claude-SearchBot, PerplexityBot, **Bingbot** | The site can't be cited in that assistant's answers. **Allow these.** Bingbot belongs here because Copilot and several assistants ground on Bing's index. |
| Training | GPTBot, ClaudeBot, Google-Extended, CCBot, Applebot-Extended, Meta-ExternalAgent | Owner's policy choice about model training; doesn't remove the site from answers |
| User-triggered retrieval | ChatGPT-User, Perplexity-User, Claude-User | Fetches when a live user asks; may not fully respect robots.txt |

The default `User-agent: * / Allow: /` robots.txt already allows all of them —
no explicit allowlist needed. Explicit stanzas only matter when the owner
wants to opt out of **training** while staying visible in answers:

```
# Stay citable, opt out of model training
User-agent: GPTBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: *
Allow: /
Sitemap: https://DOMAIN/sitemap.xml
```

Never block the search-indexing bots "for safety" — that silently removes the
site from ChatGPT/Claude/Perplexity answers.

Note for Awesomate-managed domains: the CDN prepends a managed section to the
site's robots.txt (a Content Signals preamble plus disallows for the
*training*-bot family). Search-indexing bots stay allowed, so the site remains
citable — but the served robots.txt is the site file *plus* that preamble.
Always check the live `https://DOMAIN/robots.txt`, not just the file you wrote.

## robots.txt allows it — but the CDN can still 403 it

Cloudflare **blocks many AI crawlers by default on new zones** (GPTBot,
ClaudeBot, PerplexityBot, CCBot, Google-Extended). The crawler gets a 403 or a
challenge page before it ever reads robots.txt, so a perfect robots.txt proves
nothing. Always verify from the outside:

```bash
curl -s -o /dev/null -w "%{http_code}\n" -A "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.0; +https://openai.com/gptbot)" https://DOMAIN/
curl -s -o /dev/null -w "%{http_code}\n" -A "Mozilla/5.0 (compatible; ClaudeBot/1.0)" https://DOMAIN/
curl -s -o /dev/null -w "%{http_code}\n" -A "Mozilla/5.0 (compatible; PerplexityBot/1.0)" https://DOMAIN/
```

200 = reachable. 403, 503, or an HTML page mentioning Cloudflare = blocked at
the edge. Where to fix it:

- **Awesomate-managed domain** (`*.awesomate.site` etc.): the zone is managed
  by Awesomate — the user can't change it. Raise it via `awesomate_support`
  so the zone's AI Crawl Control settings get reviewed.
- **Custom domain on the user's own Cloudflare account**: fix it with a WAF
  custom rule, action **Skip**, expression:

  ```
  cf.verified_bot_category in {"AI Crawler" "Search Engine Crawler"}
  ```

  Match on the **verified category, never on user-agent strings** — a UA
  allowlist is trivially spoofed and silently rots as bots are renamed.
  Cloudflare's own AI-bot toggle does not separate citation bots from training
  bots, which is why the toggle alone is the wrong instrument.

Verified 2026-09-02 across awesomate.ai, hub.awesomate.ai and eight live client
sites: all four retrieval bots got **200**, so this is not currently biting the
fleet. Re-check per site rather than assuming either way — it is a per-zone
setting and new zones are steered into blocking at signup.

## Most AI crawlers do not run JavaScript

**Googlebot, Gemini and Applebot render JS. GPTBot, ClaudeBot, PerplexityBot,
OAI-SearchBot and Claude-SearchBot do not** — they read the **raw HTML
response**. So a client-rendered app (React/Vue/Next SPA without SSR) is
invisible to ChatGPT, Claude and Perplexity regardless of its meta tags,
schema or llms.txt.

`awesomate_site_audit` checks this automatically (gate G1) — run it rather
than eyeballing the page in a browser, which renders the JS and hides the
problem.

Red flags — check the raw response, not the browser:

```bash
curl -s https://DOMAIN/page | head -100
```

- Body is essentially an empty `<div id="root">` / `<div id="app">` /
  `<div id="__next">`
- Full HTML under ~5KB for a content page
- The content exists only inside `<script>` tags as JSON

Fix: keep public, indexable content in static or server-rendered HTML — for
app projects, static marketing/landing pages in front of the app is usually
enough. Don't rebuild a working app over this, but never claim AI
discoverability for client-rendered content.

## The accidental opt-out — check before blaming anything else

A page can be perfectly built, reachable and fast, and still be barred from
every AI answer by one directive. Sweep for these before investigating
anything subtler:

- `<meta name="robots" content="noindex">` — out of search AND AI answers
- `nosnippet` or `max-snippet:0` — nothing may be quoted, which is the same as
  being uncitable even though the page still ranks
- an `X-Robots-Tag` response **header** carrying any of the above (easy to miss:
  it is not in the HTML, so viewing source shows nothing)
- a staging `robots.txt` that shipped to production with `Disallow: /`
- in Search Console, the Search generative AI control left off

The usual causes are an SEO plugin default, a staging file promoted by mistake,
or a "private launch" setting nobody turned back off. `awesomate_site_audit`
covers the meta and header cases (gate G3); the Search Console control has to be
checked by the owner in their own account.

Note that blocking `Google-Extended` does **not** remove a site from AI
Overviews or AI Mode — that control is about training, not answers. Don't
"fix" it expecting a visibility change.

## llms.txt rules

- Optional inventory for non-Google assistants — Google ignores it.
- Under 10KB (assistants may truncate or skip larger files).
- Markdown links: `[Title](URL): one-line description`. Only real,
  high-value pages; only true facts.
- Supplements sitemap.xml, never replaces it. Update both when content changes.