robots.txt AI Bot Checker

AI Bot Checker Quick Facts

How This Tool Works

When you enter a URL, this tool fetches the robots.txt file from the root of that domain. The robots.txt file is a plain text document that website owners use to communicate crawling rules to bots. It follows the Robots Exclusion Protocol, a standard that has been in use since the early days of the web (originally proposed by Martijn Koster in 1994). Each rule specifies a User-agent (the bot name) and a directive (Allow or Disallow) for specific URL paths.

The tool then parses every User-agent block in the file and cross-references them against nine major AI bots. It checks for both specific bot rules (like User-agent: GPTBot) and wildcard rules (like User-agent: * with Disallow: /). A wildcard disallow blocks all bots that are not explicitly allowed elsewhere in the file. The results are displayed in a clear table showing each bot's name, its purpose (training vs. search), and its current access status on your site.

Keep in mind that robots.txt is a voluntary standard. Well-behaved bots from major companies like OpenAI, Anthropic, and Google respect these rules, but not all crawlers do. The tool gives you visibility into what your declared policy is, which is the first step toward making intentional decisions about AI access to your store content.

The fetching process works through a server-side proxy to avoid CORS restrictions that would prevent direct browser-to-site requests. Your URL is sent securely, the robots.txt content is retrieved, and all parsing happens in your browser. No data is stored or logged. The entire analysis takes just a few seconds.

Step-by-Step Guide to Checking Your AI Bot Exposure

1. Enter your website URL. Type your store's domain in the input field. You can enter just the domain (example.com) or the full URL (https://example.com). The tool will automatically add the https:// prefix if needed and append /robots.txt to fetch the file.
2. Click "Check robots.txt." The tool fetches your site's robots.txt file and parses it for AI-specific rules. This typically takes 1-3 seconds depending on your server's response time.
3. Review the summary. The top-level result shows how many of the 9 AI bots can currently access your site. A high number (7-9 allowed) means you have minimal AI bot blocking. A low number (0-2 allowed) means you have aggressive blocking in place.
4. Examine the detailed table. Each row shows a specific AI bot, its purpose (training, search, or browsing), and its current access status. "Blocked" means an explicit Disallow rule exists. "Not mentioned" means no specific rule exists, so the bot is allowed by default.
5. Decide on your strategy. Based on the results, determine whether you want to block training bots, allow search bots, or make other changes. See the strategy section below for guidance.
6. Implement changes if needed. Edit your Shopify theme's robots.txt.liquid file to add or modify AI bot rules. Re-run this tool after making changes to verify they are applied correctly.

Real-World Examples

Different stores have different AI bot strategies. Here are three common configurations and the reasoning behind each.

Example 1: Open Access (Maximize AI Search Visibility)

A fashion brand selling trendy accessories decides to allow all AI bots, including training bots. Their reasoning: they want maximum exposure in AI search results, they update their product descriptions frequently so training data becomes stale quickly, and they believe the traffic benefits of AI search visibility outweigh the content protection risks. Their robots.txt has no AI-specific rules.

Example 2: Selective Blocking (Protect Content, Allow Search)

A specialty coffee roaster with detailed product descriptions and brewing guides blocks training bots but allows search bots. Their unique content is a competitive advantage, and they do not want it used to train models that could help competitors generate similar copy. But they want to appear in AI search results when customers ask questions like "best single-origin coffee beans."

Example 3: Full Block (Maximum Content Protection)

A luxury jewelry brand with proprietary product descriptions, professional photography metadata, and extensive gemstone education content blocks all AI bots. Their content represents years of expert writing and significant photography investment. They rely primarily on direct traffic, email marketing, and traditional Google search for customer acquisition. AI search is not yet a meaningful traffic source for their niche.

AI Bot Types: Training vs. Search vs. Browsing

Understanding the different purposes of AI bots helps you make an informed blocking decision. Not all AI crawlers serve the same function.

The key distinction is between training bots (which ingest your content to improve AI models) and search bots (which index your content to show it in AI search results). Blocking training bots protects your intellectual property. Blocking search bots reduces your discoverability. Many merchants choose to block training bots while allowing search bots, getting the best of both worlds.

Why This Matters for Your Shopify Store

For Shopify merchants, AI crawling has direct implications for both content protection and discoverability. If you have invested in professional product photography, detailed descriptions, or expert buying guides, that content has real value. AI training bots like GPTBot and Google-Extended can ingest this content and use it to generate responses that compete with your original work. At the same time, AI search platforms like Perplexity and SearchGPT are becoming significant traffic sources. Blocking their crawlers means your products will not appear in AI-generated shopping recommendations or product comparisons.

This is particularly relevant for stores using Rubik Combined Listings to organize large catalogs. When you have separate product pages for each color or style, each page carries unique content that could be indexed by AI search engines. A store with 50 combined product pages has 50 opportunities to appear in AI search results. Understanding which bots can access those pages helps you make informed decisions about your crawling policy, whether you want maximum AI search visibility or maximum content protection.

The financial stakes are growing. Early data from Shopify stores that appear in AI search results shows referral traffic from Perplexity and ChatGPT growing 30-50% quarter-over-quarter. While this traffic is still small compared to traditional Google search for most stores, it is the fastest-growing referral channel in ecommerce. Stores that block AI search bots today may find it harder to reclaim visibility later, as AI search algorithms develop their own ranking factors based on historical crawl data.

Common Mistakes to Avoid

• Blocking all bots with a wildcard without understanding the impact. Adding User-agent: * / Disallow: / blocks every bot, including Google's standard search crawler. This would remove your store from Google search entirely. Always use specific User-agent directives for AI bots rather than broad wildcards, unless you also add explicit Allow rules for bots you want to keep.
• Confusing Google-Extended with Googlebot. Google-Extended is specifically for AI training and AI Overviews. Googlebot is the standard search crawler. Blocking Google-Extended does not affect your regular Google search rankings. Many merchants avoid blocking Google-Extended out of fear it will hurt their SEO, but the two crawlers are completely separate.
• Forgetting to check after theme changes. Shopify theme updates, app installations, and migrations can overwrite your robots.txt.liquid customizations. Always re-check your robots.txt after any significant change to your theme or app stack.
• Assuming robots.txt provides absolute protection. Robots.txt is a voluntary standard. While major companies like OpenAI, Google, and Anthropic respect it, smaller or less reputable crawlers may ignore it entirely. For stronger protection, you need server-side blocking through a CDN like Cloudflare or a firewall that blocks specific User-agent strings at the network level.
• Not having any AI bot strategy at all. The worst approach is having no strategy. If you have never thought about AI bot access, you are making a decision by default: full access to all crawlers. Run this tool, understand your current exposure, and make an intentional choice based on your business goals.
• Blocking AI search bots without checking referral traffic first. Before blocking PerplexityBot or OAI-SearchBot, check your analytics for referral traffic from perplexity.ai, chatgpt.com, and searchgpt.com. If these sources already send you customers, blocking their crawlers will cut off that revenue stream immediately.
• Setting and forgetting your robots.txt. The AI landscape changes rapidly. New bots appear, existing bots change their behavior, and new AI search platforms launch regularly. Review your robots.txt configuration at least quarterly to ensure it reflects your current strategy and covers the latest crawlers.

Tips and Best Practices

• Separate training bots from search bots. Consider blocking training-only bots (GPTBot, Google-Extended, anthropic-ai, Bytespider) while keeping search bots (OAI-SearchBot, PerplexityBot, ChatGPT-User) allowed. This protects your content from being used as training data while maintaining visibility in AI search results.
• Review your robots.txt after every theme or app change. Some Shopify apps and themes modify robots.txt through the theme's robots.txt.liquid file. Always re-check your configuration after installing new apps or switching themes to make sure your AI bot rules are still in place.
• Use Shopify's robots.txt.liquid for customization. Since 2021, Shopify allows merchants to customize robots.txt through the robots.txt.liquid template in their theme. This is the proper way to add AI bot rules rather than relying on third-party apps or workarounds.
• Monitor AI search traffic in your analytics. Before deciding to block AI search bots, check your referral traffic from sources like perplexity.ai, chatgpt.com, and searchgpt.com. If these channels already send you customers, blocking their crawlers would cut off that traffic.
• Re-check quarterly as the landscape evolves. New AI bots appear regularly. A robots.txt file written in 2024 may not cover bots launched in 2025. Run this tool every few months to make sure your policy covers the latest crawlers and aligns with your current business goals.
• Consider partial blocking for specific paths. Instead of blocking an AI bot entirely, you can block access to specific directories. For example, blocking GPTBot from /blogs/ protects your written content while allowing it to crawl /products/ for potential AI search inclusion. This granular approach gives you more control.

When to Use This Tool

Related Free Tools

• Robots.txt Generator - Generate a custom robots.txt configuration for your Shopify store, including AI bot blocking rules. Use the checker to audit your current state, then use the generator to create an improved configuration.
• AI Readiness Checker - Evaluate how well your Shopify store is prepared for the AI-driven search era. Checks structured data, content quality, and technical factors that affect AI search visibility.
• SEO Checker - Run a comprehensive SEO audit on any Shopify page. Checks meta tags, headings, images, schema markup, and other factors that affect both traditional and AI search performance.