Can you block AI training and stay in search?

Your company may want people to find its website through an AI assistant without having its articles, research or product information used to train the next model. For some providers, those are separate choices. For others, one setting controls more than you might expect.

A Redvault scan dated 24 October 2026 illustrates why the distinction matters. Of 607 non-empty robots.txt files collected from a 1,000-domain .uk sample, 116 contained an explicit GPTBot group with a Disallow: / rule. In a separate homepage test, 44 of the 90 sites that returned a successful browser-labelled response also returned one to a request labelled GPTBot. That is a result about our test requests, not proof that the real OpenAI crawler ignored a rule or passed a site's controls.

There are four practical choices: publish a training preference, enforce access restrictions, limit AI search as well, or offer licensed access. Each addresses a different problem. None can promise that public content will never reach a model, and blocking a training crawler need not mean giving up search visibility.

Training, search and user requests are different jobs

A training crawler gathers material that may help build future models. A search crawler indexes pages so an assistant can retrieve or cite them. A user-requested fetch visits a page because someone asked the assistant to open it. The distinction is useful, but it does not mean every provider runs three completely separate bots.

OpenAI documents independent controls for GPTBot and OAI-SearchBot: a site can disallow training while remaining eligible for ChatGPT search. ChatGPT-User handles user-initiated visits, for which OpenAI says robots.txt may not apply. Allowing a search crawler makes discovery possible; it does not guarantee a citation or a visit. See OpenAI's crawler documentation.

Anthropic separates ClaudeBot, Claude-SearchBot and Claude-User. Unlike OpenAI's wording about user requests, Anthropic says its bots honour robots.txt, including the user-requested fetcher. There is no universal rule that assistants may ignore the file whenever a person asks them to browse. See Anthropic's crawler policy.

Applebot-Extended is a usage-control token, not a crawler. Applebot collects the pages; the Extended rule tells Apple whether that data may train its generative foundation models. Disallowing it does not itself remove pages from Apple's search results. Apple explains the distinction.

Google-Extended is also a control token, with no separate HTTP user agent. It governs both future Gemini training and grounding in specified Gemini products. Blocking it therefore affects more than training, although it does not affect inclusion or ranking in Google Search. Google's AI Overviews and AI Mode use separate Search controls, covered below. See Google's documentation.

A preference is useful. It is not access control.

A crawler that follows robots.txt reads the file and avoids disallowed paths. The website does not have to reject that crawler at the server for the preference to work. The Robots Exclusion Protocol explicitly separates its rules from access authorisation.

This matters when testing a block. Sending a request with the name GPTBot tests what the server does with that request. It does not reproduce a compliant crawler's decision to stay away after reading robots.txt. Nor does the name prove the request came from OpenAI. Identity checks, IP reputation, rate limits and other rules can produce different results.

Access restrictions still have a purpose: they reduce what collectors can retrieve directly from your site, including collectors that ignore preferences. But keeping pages public while distinguishing every unwanted collector from a human reader is a harder problem than maintaining a list of bot names.

What our .uk scan actually measured

The supplied sample contains 1,000 distinct .uk domains selected in rank order from the Tranco top-million list downloaded on 24 October 2026. It is a domain sample, not a census of British businesses or a direct ranking of their human readership. UK organisations using .com and other suffixes are outside it. Tranco describes its list as a research-oriented ranking.

The raw results contain 611 HTTP 200 responses with a retained robots.txt body: 607 non-empty and four empty. The remaining 389 produced no usable body. The collector excluded responses labelled as HTML and recorded failed fetches without detailed error reasons. Those 389 cannot all be described as bot blocks: they may include access restrictions, unavailable files or network failures.

The chart uses the 607 non-empty files as its denominator, matching the original analysis. It counts an explicit group for each named token containing Disallow: /. It does not apply wildcard groups or resolve every Allow exception. Seven of the 116 GPTBot groups contain non-empty Allow rules, so calling all 116 complete site-wide bans would be wrong.

Within that definition, 158 of the 607 files contained a root Disallow rule for at least one of the ten training-related tokens checked, including a legacy Anthropic token. That is 26% of the analysed files, not 26% of all UK websites. The token totals reproduced from the archive match the supplied summary. They describe explicit declarations, not every effective restriction.

The follow-up test made two sequential requests to each of the 116 homepages: one with a GPTBot/1.2 user-agent string and one with a Chrome user-agent string. Both came from an ordinary cloud server. Neither was a visit from the real bot or a JavaScript-enabled browser.

Ninety sites returned HTTP 200 to the Chrome-labelled request without either of the two Cloudflare challenge markers the script checked. Of those, 44 returned the same response classification to the GPTBot-labelled request. The other 46 returned HTTP 403 in 18 cases and HTTP 402 in 28. The remaining 26 lacked a successful browser-labelled baseline and were excluded from that comparison.

The script inspected only the first 50,000 bytes and did not save the response bodies. An HTTP 200 response without those markers is not proof that it contained the full article or even the same page. A 403 or 402 is an observed response, not proof of its cause. In particular, these results cannot tell us whether a site permits genuine, verified OpenAI traffic.

This is a one-day snapshot with one request per user-agent per homepage, no repeated trials and no test of deeper URLs. The archive also lacks the full source ranking and its permanent Tranco list identifier, so the sample's historical rank positions cannot be independently reconstructed from it alone. The raw JSON, domain list and scripts are available on request.

Option 1: publish a training preference

If your goal is to limit training while retaining discovery, start with the documented training controls for the providers you care about. Keep a small register of each token, what it controls and when you last checked its documentation. A platform's AI-blocking checkbox may write these rules for you; inspect the resulting file rather than relying on its label.

For example, the following groups express a site-wide preference against collection by GPTBot and ClaudeBot. Merge them with the site's existing rules; do not replace the whole file or assume that a training rule also manages search.

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

Check whether your existing groups contain more specific Allow rules, and test the effective rules for representative URLs. Keep the documented search crawlers available if search discovery is part of the policy. Add other providers deliberately: blocking a data archive or a mixed-purpose token can affect uses beyond model training.

Microsoft illustrates the trade-off. Its published Bing guidance says noarchive excludes indexed content from generative-model training and Bing Chat answers while preserving ordinary Bing results. Its nocache option permits more limited use, and takes precedence when both tags are present. It is therefore wrong to add noarchive as a supposedly cost-free training-only setting. Check Microsoft's documented behaviour against your intended outcome.

These preferences act through the providers that recognise them. They do not stop an unrelated scraper, guarantee a place in search, or withdraw copies already collected. Treat a new licensing or reservation signal as an additional mechanism with its own support requirements, not a replacement for documented provider controls.

Option 2: enforce access restrictions

Where direct collection is a material concern, add rules at the host, CDN or web application firewall. These can refuse identified crawler requests and record the action. Their value depends on which traffic they recognise, how the rules are ordered and whether other routes to the same content remain open.

Cloudflare's September 2026 controls distinguish Disallow AI Training from Block. The former publishes no-training preferences and allows its recognised accountable mixed-use crawlers to continue for search, while blocking other training crawlers. Block also covers mixed-use crawlers such as Googlebot, Applebot and Bingbot, so it can affect search. Cloudflare also notes that its preference sync does not yet automatically convey Bing's training preference through robots.txt. Read the setting's actual behaviour before enabling it.

Akamai likewise distinguishes training crawlers, search crawlers, and fetchers or agents. That makes a selective policy possible; it does not make every visitor's identity certain. A bot manager may combine verified identities, network information and behavioural signals. A name-based rule alone remains easy to evade.

Test permitted search traffic as carefully as blocked traffic. Review real request logs and the rule that matched, check important paths and subdomains, and monitor false positives. For content that should not be public at all, use authentication and enforce access at the application. A bot preference is not a substitute for that boundary.

Option 3: restrict AI search as well

If you also want to limit AI retrieval and answers, use the relevant search and page-level controls. This is a separate business decision. It can reduce discovery without preventing every mention, cached reference or copy elsewhere.

For Google, nosnippet prevents the page's content being used as a direct input to AI Overviews and AI Mode and removes its text snippet from ordinary results. It is not the same as noindex: the URL may still appear in search. Google also offers data-nosnippet for selected text and max-snippet to limit how much can be used. The crawler must be able to read the page to see these instructions. Google documents the available controls.

What does blocking cost? A February 2026 preprint by Hangcheng Zhao and Ron Berman found an approximately 10% short-term reduction in SimilarWeb traffic associated with publishers blocking GenAI crawlers. Its separate Comscore analysis of human browsing was negative but statistically insignificant. The longer-window estimate also lost statistical significance after about 20 weeks; that does not prove the effect disappeared. This was a publisher study of crawler restrictions, not a clean experiment on training-only opt-outs or a forecast for your business. See the paper and its limitations.

Pew Research Center found that the US adults it observed clicked traditional search results on 8% of Google visits with an AI summary, compared with 15% without one. That describes observed behaviour across different searches; it does not tell us what would happen if an individual publisher blocked a bot.

We'd measure qualified visits, enquiries and conversions from the relevant channels before changing access, then monitor them afterwards. Mentions, citations, visits and sales are different outcomes. A provider citing a site that disallows a training bot is not, on its own, evidence that the training rule failed.

Option 4: offer licensed access

Licensing can sit alongside restrictions. The useful question is whether a buyer wants your material on terms that justify supplying it, not whether your website can display a payment signal.

Cloudflare's Pay Per Crawl, documented as a closed beta, lets participating site owners set a price for crawler access. A request without the required payment intent can receive HTTP 402 with pricing. This needs a participating crawler and working commercial arrangements; it does not make every scraper pay.

Really Simple Licensing defines machine-readable terms in an RSL document, which can be associated with content through mechanisms including robots.txt. The declaration and the licensing transaction are distinct. Publishing terms does not itself produce demand, agreement or revenue.

For a company with valuable specialist material, we'd assess the rights it can license, likely buyers, delivery method, permitted uses and enforcement costs. We would not assume only large newspapers can earn licensing income, or budget for payments merely because a standard supports them.

What a new block cannot undo

A new rule cannot retroactively prevent a fetch that has already happened or, by itself, change a trained model. That is different from saying every page is already in every dataset, or that no removal or legal remedy can ever be available.

Common Crawl says its published archive files are immutable, but also describes filtering affected URLs from later crawls and making them inaccessible through its public tools and indices in response to removal requests. Its explanation does not promise deletion from copies already distributed. See Common Crawl's removal policy explanation.

Copies need separate attention. Content supplied to partners, republished elsewhere or available through an unprotected API may remain reachable under different controls. An authorised reader can also copy or upload what they can see. Reduce unnecessary bulk access, review distribution terms and keep sensitive material behind genuine access controls.

We would not treat image-protection tools, crawler traps or browser challenges as universal solutions. They have different aims and can affect availability, accessibility or discovery. Evaluate a specific tool against the threat and the audience instead of adding it because it is labelled anti-AI.

The legal position is not one global opt-out

The distinction between an access rule and a legal right matters. A robots.txt directive can communicate intent; whether copying is lawful depends on the applicable law, the material, the use and any licence or exception. The following is a practical overview as of 24 October 2026, not a ruling on a particular collection.

In the UK, the specific text-and-data-mining exception in section 29A is limited to non-commercial research with lawful access. It is not a general permission for commercial AI training. Protected copying normally requires authorisation unless an exception applies, but rights, territorial issues and the facts still matter. The government's March 2026 report said a broad exception with an opt-out was no longer its preferred approach; that policy statement did not itself change the law. See the government's report.

In the EU, the general text-and-data-mining exception depends on conditions including lawful access and the absence of an appropriate rights reservation. Online reservations need suitable machine-readable implementation. The separate scientific-research exception has different rules; a reservation is not a veto over every permitted use. The EU's explanation sets out why a reservation can require a provider to seek authorisation.

The AI Act adds copyright-policy obligations for general-purpose model providers. The Commission says these apply to models placed on the market from 2 August 2025, with a transition until 2 August 2027 for models already on the market before that date. The voluntary Code of Practice is one compliance route, not a new copyright right. See the Commission's timetable.

In the US, there is no single statutory AI-training opt-out comparable to that EU reservation mechanism. Fair use is a statutory limitation on copyright, assessed through factors including purpose, the work, the amount copied and market effects. It is inaccurate to say there is no exception, or that all training is automatically lawful. Section 107 sets out the test. A bot rule alone does not determine its outcome.

If protecting or licensing the content is commercially important, get advice on the rights you hold and the jurisdictions involved. Technical controls and dated records of your preferences can support that work; they do not replace it.

What we'd do

First, decide which outcome matters for each part of the site. A public product page, a paid research archive and an internal knowledge base need different policies. Write down what may be used for training, what should remain discoverable and what requires a signed-in customer.

Next, implement the relevant provider preferences and review any existing platform switches. Check what the deployed robots.txt and page headers actually say, including conflicts and exceptions. Keep a dated copy so a later change can be traced.

Where access needs enforcement, configure it at the appropriate host or edge service and check the logs. Confirm both sides of the policy: intended requests are stopped and wanted readers and search crawlers still work. A synthetic user-agent test is one diagnostic, not the whole acceptance test.

Then measure the business effect. Review search visibility, useful referrals, conversions and support complaints, and revisit the settings when providers change their controls. Explore licensing where there is a credible buyer or a low-cost experiment, without treating expected revenue as a certainty.

The decision is about how much access to offer, for which uses, and at what cost. A cybersecurity assessment can review the controls already in place, identify gaps and turn those choices into a prioritised action plan. Copyright advice and any implementation work need their own scope.