Your website is being asked to serve two different jobs

If you run a local service company, consultancy, shop or professional practice, the question is rarely “Do I want AI on my website?” The real question is which AI system gets access, for what purpose, and what you receive in return.

A search crawler may help your business appear when somebody asks Google, ChatGPT or another answer engine for a recommendation. A training crawler may collect your pages for a future model without sending a visitor, lead or sale back. Those are different exchanges. Treating them as one is the mistake.

My default recommendation for a small business is straightforward: keep Googlebot and other genuine search crawlers accessible, allow answer-engine crawlers when you want citations and referrals, and make a separate decision about training crawlers. If your content is ordinary service information that already exists to explain what you do, allowing discovery is usually worth testing. If it contains a proprietary calculator, paid research, a detailed process, original templates or a hard-won database, training access deserves a stricter rule.

That is the same ownership principle behind A Useful Calculator Still Wins When AI Can Reproduce the Answer. Give away enough information to earn the enquiry. Protect the asset that makes your business difficult to replace.

Search access and training access should not be judged together

The important distinction is in the crawler names

Google makes the split unusually clear. Googlebot controls crawling for Google Search, including eligibility for links in AI Overviews and AI Mode. Google-Extended is a separate product token for managing whether content may be used to train future Gemini models. Blocking Google-Extended is therefore not the same as blocking Google Search. Google explicitly says that robots.txt rules for Googlebot are the control for Search, while Google-Extended governs other uses. (developers.google.com)

OpenAI uses a similar separation. Its publisher guidance says OAI-SearchBot helps content appear in ChatGPT search, while GPTBot is the user agent publishers should disallow when they want to exclude pages from possible training. OpenAI also says that ChatGPT referrals can be tracked because referral URLs include a `utm_source=chatgpt.com` parameter. ([help.openai.com](https://help.openai.com/en/articles/12627856-publishers-and-developers-faq%23%3A~%3Atext%3DChatGPT%20automatically%20includes%20the%20UTM%20parameter%20utm_source%3Dchatgpt.com%20in%20referral%20URLs%2C%20enabling%20clear%20tracking%20and%20analysis%20of%20inbound%20traffic%20from%20ChatGPT%20search%20results?utm_source=openai))

Anthropic separates ClaudeBot, which it describes as collecting public web content that could contribute to model training, from Claude-User, which retrieves pages in response to a user's request. A business could reasonably allow the latter while restricting the former. Anthropic says both respect robots.txt and that ClaudeBot also supports Crawl-delay. (support.anthropic.com)

This is the practical answer to “Should I create different rules for search engines and answer engines?” Yes, where the operator provides distinct crawlers. Do not assume that blocking a training bot blocks the answer product, or that allowing an answer bot means you have agreed to model training.

The numbers argue against giving blanket permission

The traffic case is real, but it is not large enough to justify surrendering control

There is evidence that AI referrals are becoming useful. An analysis of 391 small and medium-sized business websites found generative-AI referral traffic rose by 123 percent between September 2024 and February 2025. That does not mean every small business will see a meaningful stream of leads, but it does make a total block harder to defend when answer engines are already influencing discovery. (searchengineland.com)

The opposing evidence is just as important. Cloudflare reported that, in its June 2025 data, OpenAI crawled at a ratio of roughly 1,700 requests for every referral and Anthropic at roughly 73,000 to one. Its later analysis reported an OpenAI ratio of 1,091 crawls per human referral in July. These figures are not a forecast for your site. They are a warning that crawler volume and business value are not the same thing. (blog.cloudflare.com)

That distinction matters for a small business with limited hosting, a thin margin or a site full of expensive generated pages. If a crawler consumes resources, copies the useful part of your explanation and sends no visitor back, “more visibility” is not automatically a return on investment.

Google search adds another complication. Independent studies disagree on the size and direction of click loss from AI Overviews because they use different keywords, devices, time periods and datasets. Semrush’s analysis of more than 200,000 keywords found that keywords with AI Overviews had higher zero-click rates, but that the rate was not rising in its measured period. Ahrefs later reported a much sharper reduction in click-through rate for the top organic result. Neither figure should be pasted into your forecast without checking your own Search Console data. (semrush.com)

The sound business response is measurement, not panic. Keep the access that can create discovery. Restrict the access that only creates extraction. Then watch enquiries, branded searches, assisted conversions and referral sessions rather than bot requests alone.

A workable crawler policy for a small business

1. Map the crawler purpose

   List the user agents your site sees and classify each as search, answer retrieval, training, advertising validation or unknown.

2. Keep ordinary search access open

   Do not block Googlebot or Bingbot if organic search remains an important source of enquiries. Google warns that robots.txt is not a way to hide a page from search results.

3. Separate answer access from training access

   Allow OAI-SearchBot, Claude-User or comparable retrieval crawlers only when you want the associated answer product to discover and cite your pages.

4. Protect the valuable paths

   Use stricter rules for paid resources, customer portals, calculators, original datasets and pages that expose a repeatable competitive advantage.

5. Check the edge layer

   Test the actual response from the crawler. A permissive robots.txt file does not help if a CDN, WAF, CAPTCHA or bot-management rule returns a forbidden response.

6. Review the policy after real data arrives

   Compare AI referrals, assisted conversions, server load and lead quality. Change access when the exchange proves unbalanced.

Robots.txt is a signal, not a lock

This is where many small businesses overestimate what they have configured. Robots.txt expresses a crawler preference. It does not authenticate the visitor, enforce a licensing agreement or guarantee that an unknown scraper will obey. Google itself says not to use robots.txt as a method for hiding pages from search. (developers.google.com)

Research presented at IMC 2025 found that 14 percent of the top 10,000 sites studied actively blocked two Anthropic crawlers, while the authors also described robots.txt as a blunt instrument with unclear boundaries around what a “robot” is. In other words, a rule in the file is useful for cooperative crawlers, but it is not a complete access-control system. (people.cs.uchicago.edu)

If access matters, enforce it at the CDN, WAF or application layer. Cloudflare’s documentation distinguishes between managed robots.txt preferences and AI Crawl Control, which can track crawler activity and apply blocks. OpenAI’s own troubleshooting guidance also tells site owners to check WAFs, CDNs, bot mitigation, authentication, JavaScript challenges and rate limits because those layers can block legitimate crawlers even when robots.txt allows them. (developers.cloudflare.com)

Do not create an elaborate policy before you know whether the problem exists. Look at access logs, identify the user agents consuming resources, and test a representative page. If the only thing you discover is occasional retrieval from an answer engine that sends qualified visitors, leave it alone. If a training crawler is repeatedly fetching thousands of pages and producing no measurable value, block it or rate-limit it.

Make the policy fit the business, not the internet argument

A plumber with service pages, locations and clear pricing guidance should usually prioritize being found. A specialist consultancy with a distinctive diagnostic method may want answer engines to understand its expertise while keeping its paid framework and detailed templates out of training datasets. An ecommerce business may allow product discovery but restrict internal search, customer data and low-value faceted URLs.

That is why “allow everything” and “block everything” are both weak policies. The first gives away access without checking the exchange. The second can remove a new discovery channel before the business has measured it. The better approach is to separate purposes, write rules by crawler and path, and revisit them when the numbers change.

Your website should remain useful to machines that can help a customer find you. It does not need to become a free, unrestricted input stream for every system that can request a page. That is a commercial decision, and small businesses are entitled to make it deliberately.