Guide

How to configure robots.txt for AI agents

AI assistants like ChatGPT, Claude and Perplexity use their own bots (GPTBot, ClaudeBot, PerplexityBot) to read websites. If your robots.txt blocks them, assistants cannot see your shop and cannot recommend it to customers.

Which bots to check

Open robots.txt on your website (e.g. yourdomain.com/robots.txt). Look for User-agent blocks for these names: GPTBot, ChatGPT-User, ClaudeBot, claude-web, PerplexityBot, Google-Extended, Amazonbot, Applebot-Extended. If they are followed by Disallow: /, the bot is fully blocked.

How to unblock AI bots

Remove or comment out the Disallow: / lines for each AI bot. If you want to block only certain parts of the site (e.g. admin), keep Disallow only for those specific paths:

User-agent: GPTBot
Disallow: /admin/
Disallow: /checkout/

User-agent: ClaudeBot
Disallow: /admin/
Disallow: /checkout/

After saving, re-check your site at idealbusiness.ai.

Shoptet

In Shoptet admin, go to Settings → SEO → robots.txt. Find the rules for GPTBot, ClaudeBot and other AI bots and remove the Disallow: / lines. Shoptet adds some blocks automatically — after saving, verify they haven't returned.

Shopify

Since June 2024 Shopify no longer blocks AI bots. If you have older customisations in robots.txt.liquid, check them: Online Store → Themes → Edit code → Templates. Remove any Disallow rules for GPTBot and ClaudeBot.

WooCommerce and WordPress

WordPress generates robots.txt virtually. Install Yoast SEO or Rank Math, find the robots.txt section in settings and remove any rules blocking AI bots. If there are none, check whether a physical robots.txt file exists on the server.

Check your website — find out what AI assistants see on your pages.