AI crawlers and robots.txt
Allow Claude, Gemini, ChatGPT, and Perplexity to discover OpenQuok public pages when Cloudflare managed robots.txt is enabled.
Connect your agent today
Draft from chat, review in your calendar, and publish only what you approve.
Overview
The web app exposes /robots.txt, /llms.txt, and /llms-full.txt so bots can find public docs and marketing pages. Sign-in and workspace URLs stay off-limits.
/mcp on the website is docs-only (no API key). Workspace automation uses /mcp on the **API** — same path, different host.If an AI visibility checker says Claude or Gemini cannot see you, read live /robots.txt first. On Cloudflare, turning off block training in robots.txt (managed robots) fixes this more often than changing OpenQuok settings alone.
Why “Training → Allow” is not enough
Cloudflare has two separate controls:
| Control | What it does | Claude / Gemini |
|---|---|---|
| Training → Allow (do not block) under Configure AI bot policies | Stops Cloudflare from HTTP-blocking training crawlers at the edge | Necessary, but not sufficient |
| Set your preference to block training in robots.txt (managed robots.txt) | Prepends Disallow: / for ClaudeBot, Google-Extended, GPTBot, … | What robots-based visibility checkers see |
Warning
curl -sS https://www.openquok.com/robots.txt still shows # BEGIN Cloudflare Managed content with ClaudeBot / Google-Extended and Disallow: /, managed robots.txt is still on — regardless of Training Allow.Fix in Cloudflare (production)
Open Security Settings → Bot traffic
In the Cloudflare dashboard, select the zone that serves www.openquok.com. Open Security → Settings, filter by Bot traffic (or follow the here link from the Training policy copy).
Turn OFF managed training blocks in robots.txt
Find Set your preference to block training in robots.txt (managed robots.txt) and turn it off. Do not stop at Training → Allow (do not block) — that is a different setting.
That is the control that prepends Disallow: / for ClaudeBot, Google-Extended, GPTBot, and related training crawlers. With it off, crawlers see only the OpenQuok origin file (Content Signals + auth disallows + explicit AI Allow groups). OpenQuok still emits ai-train=no via Content-Signal, so you keep a soft training preference without a site-wide crawl block.
Or flip it via API
Create a Cloudflare API token with Bot Management Write for the marketing zone, then:
export CLOUDFLARE_API_TOKEN='…'
export CLOUDFLARE_ZONE_NAME='openquok.com' # or set CLOUDFLARE_ZONE_ID
pnpm --filter ./web run fix:ai-robotsThis sets is_robots_txt_managed to false on the zone (see Update Zone Bot Management Config).
Keep Training Allow (already correct if set)
Under Security → Settings → Configure AI bot policies, leave Training on Allow (do not block). That only affects edge HTTP blocking.
Optional: AI Crawl Control Allow
Under Security → AI Crawl Control, set Action to Allow for crawlers you want (at least ClaudeBot and Google-Extended). This also controls WAF blocking.
Verify the live file
pnpm --filter ./web run verify:ai-robots
# or
curl -sS "https://www.openquok.com/robots.txt"Pass criteria:
- No Cloudflare managed section with ClaudeBot / Google-Extended and Disallow: /
- Origin suffix still lists Sitemap: and Allow: / for those bots
After robots is fixed: AI crawlers may still lag
When pnpm —filter ./web run verify:ai-robots passes, bots are allowed in,but Claude or Gemini may still not show your site for days or weeks. Each engine crawls and updates on its own schedule.
More robots.txt changes will not speed that up. Keep docs, pricing, compare, and /llms.txt public, and use Cloudflare AI Crawl Control to confirm ClaudeBot and Google-Extended are actually hitting your domain.
What the web app emits
The route web/src/routes/robots.txt/+server.ts builds:
- User-agent: * — Content-Signal (search=yes, ai-input=yes, ai-train=no) plus disallow auth, editor, admin, and OAuth paths only
- Per–AI-bot groups with Allow: / plus the same auth/admin disallows (including Claude-SearchBot / Claude-User)
- Meta crawler allows (Facebook / Instagram app verification)
- Comments pointing to /llms.txt and /rss.xml
Every public HTML page also advertises the LLM index via <link rel="alternate" href="/llms.txt"> in global meta tags (createMetaData).