Robots.txt can influence whether certain AI crawlers access a website, but it is not a universal switch for removing content from AI search, Google results or AI model knowledge. It is a set of crawl instructions placed at the root of a domain. Its effect depends on which bot is visiting, whether that organisation honours the instructions, and whether the content has already been indexed, cached, syndicated or used elsewhere.
For website owners, the practical question is not simply whether to block “AI”. It is whether you want to permit or restrict access for search crawlers, AI search crawlers, model-training crawlers, user-triggered browsing tools and data providers. Those choices can affect discoverability in Google Search, Google AI Overviews, ChatGPT, Gemini, Perplexity and other answer engines, but they can also create unintended visibility problems if applied too broadly.
This guide explains how robots.txt AI search controls work, what they can and cannot achieve, and how they fit into a wider reputation-management and AI-search visibility strategy.
What is robots.txt and what does it do?
A robots.txt file is a publicly accessible text file hosted at the root of a website, usually at https://www.example.co.uk/robots.txt. It uses the Robots Exclusion Protocol to tell compliant crawlers which areas of a site they should not crawl.
For example, a website might use robots.txt to discourage crawlers from visiting internal search pages, private document directories, staging areas or duplicate URL parameters. The file can apply instructions to all crawlers or to named user agents, such as Googlebot or a specific AI crawler.
Robots.txt is primarily a crawl-management mechanism. It is not a content-removal tool, a privacy tool or a guaranteed indexing control.
What robots.txt does not do
Website owners often overestimate what a robots.txt file can achieve. A disallow instruction does not reliably:
- remove an already indexed URL from Google, Bing or another search engine;
- delete a page from an AI model’s existing training data or prior outputs;
- remove copied, syndicated or quoted content from third-party websites;
- prevent someone from visiting a public page in a browser;
- make confidential information private if the page remains publicly accessible;
- require every bot operator to comply with the instruction.
If sensitive material is publicly available, robots.txt should not be treated as access control. Proper authentication, server-side restrictions, removal of the underlying file or page, and appropriate HTTP status codes are more suitable protections.
Why AI crawlers need different treatment from traditional search bots
Traditional search engines crawl pages to index and rank them in search results. AI systems may access web content for several different purposes, and those purposes are not always controlled by the same crawler.
A single provider may operate separate bots for search indexing, AI answer generation, model training, user-initiated browsing and product-specific retrieval. Blocking one bot may therefore have no effect on another.
The main types of AI-related web access
- Search indexing: a search engine crawls pages to include them in conventional search results.
- AI search retrieval: an answer engine accesses or retrieves webpages to formulate a current response, citation or summary.
- Model training: content may be collected to help train or improve future models, subject to the provider’s policies and controls.
- User-initiated browsing: an AI assistant may access a site after a user asks it to browse, summarise or research a page.
- Monitoring and security: crawlers may visit for technical, quality, safety or abuse-prevention purposes.
These distinctions matter because a business may want to be visible in Google Search and AI search while restricting use for model training. Another organisation may decide that its original research, premium editorial material or commercially sensitive content should not be available to particular AI crawlers, even if it remains searchable on Google.
Which AI bots can be controlled in robots.txt?
Named user-agent controls are available for some AI providers, but bot names, purposes and implementation policies can change. Website owners should always consult the current documentation published by the relevant provider before changing crawl directives.
Examples that have been publicly used by major providers include:
- GPTBot: associated with OpenAI web crawling for model-training-related purposes.
- OAI-SearchBot: associated with OpenAI search features and search-result retrieval.
- ChatGPT-User: associated with user-triggered requests in which ChatGPT visits a page on a user’s behalf.
- Googlebot: Google’s primary web crawler, relevant to Google Search and Google’s search-based experiences.
- Google-Extended: a product token Google has made available for publishers seeking more control over certain Gemini and generative-AI uses. It is separate from Googlebot.
- PerplexityBot: associated with Perplexity’s crawling activities.
- Bingbot: Microsoft’s search crawler, which may be relevant to Bing Search and search-grounded Microsoft experiences.
The presence of a bot name in a robots.txt file is not proof that it will deliver a particular commercial or reputational outcome. Equally, blocking an AI crawler can reduce the likelihood of direct crawling without resolving information that has already been published, indexed or repeated by other sources.
How robots.txt directives for AI search work
Robots.txt directives are organised into groups. Each group identifies a user agent and then specifies paths that the crawler is allowed or disallowed to access.
A simplified example may look like this:
User-agent: GPTBot Disallow: / User-agent: OAI-SearchBot Allow: / User-agent: * Disallow: /private/
In this example, the site asks GPTBot not to crawl any part of the site, permits OAI-SearchBot to crawl it, and asks all crawlers to avoid the /private/ directory. The exact result depends on the bot recognising and complying with the directives.
Common implementation errors
Small technical mistakes can have large visibility consequences. Common issues include:
- placing the file in a subfolder rather than the domain root;
- using an incorrect user-agent name;
- blocking Googlebot unintentionally while trying to restrict AI use;
- creating conflicting rules across HTTP and HTTPS, www and non-www versions of a site;
- blocking important product, service or newsroom pages that support search visibility;
- assuming that a robots.txt change removes existing search results;
- using robots.txt to hide pages that should instead be removed or protected by login controls.
Before making substantial changes, test the file, check server logs where available, and monitor Google Search Console and other relevant reporting tools. A crawl restriction that looks sensible on paper can unintentionally reduce organic traffic, limit indexing of replacement content or restrict the pages that establish a business’s authority.
Can robots.txt block content from Google AI Overviews?
Not in the simple sense that many people expect. Google AI Overviews are closely connected to Google Search’s crawling and indexing systems. Google has stated that pages do not generally require special technical markup or separate AI files to be eligible for search features such as AI Overviews.
If a website blocks Googlebot, it may prevent Google from crawling relevant pages and can harm visibility in conventional Google Search as well as search-based AI experiences. Conversely, allowing Googlebot does not guarantee that a page will appear in an AI Overview, receive a citation or rank prominently in Google Search.
Google-Extended is distinct from Googlebot and may be relevant to publishers assessing whether their content can be used for specified Gemini and generative-AI purposes. However, website owners should avoid assuming that this control governs every Google product, every AI-generated answer or all historical use of material. Provider documentation and product policies should be checked carefully before relying on a directive.
Can robots.txt remove a negative article from AI search?
No. If a negative article appears on a publisher’s website, only the publisher can remove or substantially change the source page. The affected person or business cannot use their own robots.txt file to control another website’s content.
Even where the publisher adds a robots.txt restriction, the article may remain visible in search engines if it was previously indexed, linked to by other sites or reproduced elsewhere. AI systems may also rely on other sources that describe the same allegation, event or dispute.
This is particularly important for executives, professionals and businesses affected by negative press, false allegations, outdated reporting or harmful search results. The appropriate response depends on the source, the accuracy of the material, the jurisdiction, the platform’s policies and the visibility problem itself.
A reputation strategy may involve:
- source removal: engaging with the publisher or platform to seek removal, correction, anonymisation or updating where grounds exist;
- search-engine de-indexing: seeking removal of a URL from particular search results where relevant policies or legal rights apply;
- Right to Be Forgotten considerations: assessing whether UK or European data-protection principles may be relevant to name-based search results;
- search-result suppression: improving the prominence of accurate, useful and authoritative content so damaging pages are less visible for relevant searches;
- AI-search optimisation: improving the clarity, authority and corroboration of content that AI search systems may retrieve or cite.
These are different remedies. A page can be removed from a publisher’s site, de-indexed by a search engine, displaced in rankings or still mentioned in AI-generated answers depending on the circumstances. For a fuller explanation of those options, see our guide to repairing an online reputation after negative search results.
Removal, de-indexing and suppression are not the same
Confusing these approaches can lead to poor decisions, particularly where a business is under pressure to act quickly.
Publisher or source removal
Source removal means the content is taken down, amended or made inaccessible by the website that hosts it. This is generally the most complete outcome, but it depends on the publisher’s decision, legal obligations, editorial policies and the facts of the case. It may not remove copies, archives or third-party reporting.
Search-engine de-indexing
De-indexing means a search engine removes a page or name-based result from some search results. The source page may still exist and be accessible directly. Search-engine decisions are platform-specific: removal from one search engine does not automatically remove the result from another.
Search-result suppression
Suppression is a visibility strategy. It aims to improve the position of strong, relevant and legitimate content so that negative results appear lower for searches of a name, brand or topic. It does not erase the underlying page and should not be presented as a substitute for removal where removal is achievable.
For persistent negative coverage, a carefully planned negative Google search-result suppression strategy may be more realistic than pursuing a single technical fix. The same principle increasingly applies to AI search, where the quality and authority of sources can influence what a system retrieves or cites.
How robots.txt affects AI-search visibility and reputation repair
Robots.txt decisions can support or undermine reputation repair because public web content remains central to how search engines and AI systems understand people, brands and organisations.
For example, a company seeking to improve its visibility may publish authoritative pages explaining its services, leadership, policies, expertise, research or community work. If those pages are accidentally blocked from major search crawlers, the company reduces the pool of first-party evidence available to search systems.
Strong reputation content is not simply promotional copy. It should provide genuine information, have a clear purpose, identify the relevant organisation accurately and be supported by credible external signals where appropriate. Useful third-party coverage, industry listings, professional profiles, earned media and accurate references can help corroborate entity information.
AI-search optimisation, sometimes described as AEO or GEO, focuses on making content understandable and useful for answer engines as well as conventional rankings. Clear writing, direct answers, consistent organisation names, authoritative sources and well-structured pages all matter. None of these factors guarantees a citation in ChatGPT, Gemini, Perplexity or Google AI Overviews, but they can make legitimate content easier to interpret and retrieve.
Should your website allow AI crawlers?
There is no universal answer. The right approach depends on your commercial model, content type, audience, intellectual-property concerns, privacy obligations and search-visibility objectives.
Reasons a business may allow AI search crawlers
- It wants useful service, product or editorial pages to be discoverable in AI-powered search.
- It sees AI search as an emerging source of qualified enquiries and brand discovery.
- It publishes original expertise that benefits from being retrieved alongside conventional search results.
- It wants accurate first-party information to be available when people research its brand, leadership or services.
Reasons a business may restrict some AI crawlers
- It wants to limit use of original material for model training where controls are available.
- It operates a paid-content, research or licensing model.
- It has technical concerns about crawl volume or server resources.
- It needs to limit crawler access to specific low-value, duplicate or sensitive sections of the site.
- It wants to separate visibility in search from permissions for particular generative-AI uses.
The most sensible approach is often selective rather than absolute. A business might permit search crawlers to access public service pages while disallowing low-value internal paths, gated assets or content that should never have been publicly available.
A practical decision process before changing robots.txt
- Define the problem precisely. Is the concern AI model training, AI answer citations, Google indexing, unauthorised copying, negative content or confidential material?
- Identify the source. Establish whether the relevant content is on your own website, a third-party publisher, a social platform, an archive or a search-result page.
- Check existing visibility. Review search results, AI answers where relevant, crawl data, indexed URLs and referrals before making changes.
- Separate crawler types. Do not block Googlebot, Bingbot or other search crawlers merely because you want to restrict a separate AI-training crawler.
- Use the right control. Consider removal, authentication, noindex directives, canonicalisation, publisher engagement or robots.txt according to the actual issue.
- Monitor the outcome. Crawl and indexing changes can take time, and AI-search behaviour may vary by platform and query.
For people dealing with false or misleading material, the technical crawl question is usually secondary to the source and legal or policy position. Reputation Ace can help assess the distinction between source removal, search-result de-indexing and visibility work, including cases involving false accusations in search engines.
Frequently asked questions
Does robots.txt stop AI from using my website content?
Robots.txt can ask named, compliant AI crawlers not to access specified pages, but it does not create a universal legal or technical barrier. It also cannot reverse access, indexing or model training that may have occurred before the rule was added.
Will blocking GPTBot remove my content from ChatGPT?
No. Blocking GPTBot may affect future crawling by that user agent, but it does not guarantee removal from ChatGPT responses. Information may have been obtained earlier or may be available through other public sources, search retrieval or user-provided links.
Can I block AI crawlers but remain visible on Google?
Often, yes. You can use separate robots.txt groups for specific AI user agents while continuing to allow Googlebot. However, you should verify the current bot documentation and test your configuration carefully to avoid blocking search crawlers unintentionally.
Does Google-Extended block Google AI Overviews?
Google-Extended is separate from Googlebot and is intended to give publishers control over specified Gemini and generative-AI uses. It should not be treated as a guaranteed method of preventing appearance in Google AI Overviews or all Google AI experiences.
Can robots.txt remove negative news articles from Google or AI search?
No. Robots.txt on your own website cannot control a negative article hosted by another publisher. Possible options may include publisher engagement, removal requests, de-indexing assessments, correction requests and search-result suppression, depending on the facts.
Should I use robots.txt or noindex to remove a page from Google?
For a page you control, a noindex directive is usually more appropriate where the objective is to prevent indexing, provided search crawlers can access and process it. Removing the page or returning an appropriate HTTP status may be better for material that should no longer exist. Robots.txt alone can prevent crawling without reliably removing an already indexed URL.
Can AI-search optimisation guarantee citations in ChatGPT, Gemini or Perplexity?
No. AI-search optimisation can improve the clarity, usefulness and discoverability of content, but no agency can guarantee citation, inclusion or prominence in an AI-generated response. Each platform uses its own systems, sources and retrieval processes.
Get advice on AI search, search visibility and online reputation
Robots.txt is one useful technical control, but it is rarely the complete answer to an AI-search or reputation problem. Reputation Ace is a UK online reputation management and AI-search optimisation company that helps businesses and individuals assess their visibility across traditional search engines and AI-powered search.
Depending on the circumstances, the right combination may involve source removal, search-engine de-indexing, search-result suppression, authoritative content strategy, SEO, AEO, GEO or AI-search optimisation. To discuss your situation with Reputation Ace, call 0800 088 5506 or email info@reputationace.com.