Write a robots.txt That Welcomes AI Crawlers

Act as a technical SEO consultant. Generate a complete, paste-ready robots.txt for a small business site that wants AI answer engines to find and cite it.

Site: domain [YOUR-DOMAIN.COM] · sitemap URL [https://YOUR-DOMAIN.COM/sitemap.xml] (blank for placeholder) · paths to block from all crawlers [LIST or NONE] · paths to block from AI training but allow for search [LIST or NONE] · crawl delay seconds [BLANK or NUMBER]

Requirements:
1. Allow well-behaved crawlers by default
2. Explicit User-agent allow groups, each under a one-line comment naming its purpose: AI search crawlers (OAI-SearchBot, PerplexityBot, Claude-SearchBot); user-request agents, which fetch a page when someone asks an assistant about it (ChatGPT-User, Perplexity-User, Claude-User); search engine crawlers (Bingbot, Applebot); AI training controls plus the open archive crawler, which is where my block-from-AI-training paths go (GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot)
3. Sitemap directive
4. Header comment with site name and last-updated date
5. Valid syntax — one directive per line, no trailing whitespace
6. One group per crawler: a crawler obeys only the group that names it and ignores the * group once it has a group of its own, so repeat every block-for-all path as a Disallow line inside every named group

Output:
1. The robots.txt in a single code block, nothing else inside it
2. Deploy notes — where to upload, a curl command plus the robots.txt report and the URL Inspection tool in Google Search Console to verify, a one-line explanation of any block, and which AI crawlers were not included and why I might add them later

Success means the file passes a syntax check as written, my block lists appear exactly as I gave them, and every block-for-all path appears inside every group.

Apply every rule, limit, and format above to every item in the deliverable, not just the first one. Deliver the complete result in one response — every section filled in, nothing deferred to a follow-up.

If a required input is missing, ask for it once before starting; otherwise proceed and label any assumption you made.

Notes: R40 refresh; tuned to vendor prompting guidance