# OAI-SearchBot and GPTBot: choose search access separately from training access

OpenAI documents separate robots.txt controls for search discovery and model-training crawling; choose each policy independently.

Published: 2026-11-01

Updated: 2026-11-02

Author: Staff Writers

Publisher: Digital Publishing News

Earlier version reviewed by Shaun Connell, Managing Editor

Canonical: https://digitalpublishingnews.com/articles/openai-search-training-crawlers/

## Choose separate rules for search and training crawlers

Allowing a search crawler and allowing collection for model training are different choices. OpenAI, the company behind ChatGPT, documents separate robots.txt controls for OAI-SearchBot and GPTBot. You can express different rules for each rather than treating every OpenAI crawler as one visitor.

A crawler is software that retrieves web pages automatically. OAI-SearchBot supports search discovery; GPTBot is associated with content that may be used to train models. Training is the process of developing the model, rather than looking up a page for a current search result. The example below allows the search crawler and disallows GPTBot.

Source: https://developers.openai.com/api/docs/bots

## Separate search and training controls

OAI-SearchBot: Search discovery.

GPTBot: Training crawling.

Access check: Verify robots.txt and firewall behavior separately.

Diagram: Digital Publishing News, based on the documentation cited below.

## Example: allow search crawling, decline training crawling

If this is your chosen policy, merge these groups into the existing robots.txt at the site root. Review existing groups for these names before editing; do not replace unrelated crawler rules with this example. Private content still needs actual access control.

```text
User-agent: OAI-SearchBot
Allow: /

User-agent: GPTBot
Disallow: /
```

## Check the rule and the actual response

Retrieve the public robots.txt and compare it with the intended policy. Then ask the firewall operator to examine access by crawlers verified against OpenAI’s published identity information. A visitor can copy a bot name. A 403 response shows access was refused even if robots.txt permits it.

A successful page response establishes delivery, not inclusion in ChatGPT results. These groups govern the named crawlers; they do not establish what happened to previously collected content or control every user-initiated request. Keep the old file and the reason for the choice with the change record.

## Turn a crawler policy into two observable checks

Write down each use your publication intends to allow and the public paths it applies to. Then check the live robots file and the response from a representative page, separately. A rule can be correct while a firewall still blocks access. A successful browser visit from your own network also does not establish what the service receives. Record where each layer is configured so a later hosting change does not quietly alter the policy.

Use a harmless public article for the test. Compare a URL that should be accessible with a route whose policy is intentionally different. Do not use private content as a crawler-policy test. If the team later changes its training preference, rerun both checks rather than assuming the search policy survived. Keeping the two intentions in the change record helps an editor distinguish a deliberate choice from a side effect of replacing the whole file.

## Compare policy intent with the site that serves the article

Suppose you allow search discovery on one publication while you disallow training crawls. Your hosting provider migrates the site to a new domain or replaces a shared robots file. Check the new public origin instead of relying on the old configuration. Keep a dated copy of the rules on both origins and record which one serves the representative article.

If the search rule is still correct but the page response changes to an error, investigate infrastructure before editing the crawler policy. If the public page works but an unintended crawler rule changed, restore the intended distinction. Recording the two outcomes separately gives your team a clean rollback target and avoids undoing the decision it meant to keep.

Test the published policy separately on every origin that serves articles.

AI production disclosure: https://digitalpublishingnews.com/how-we-use-ai/

## Sources

https://developers.openai.com/api/docs/bots

Editorial status: AI-assisted revision; preceding edition reviewed by Shaun Connell. Current revision awaits review.