OpenAI’s separate OAI-SearchBot is used for ChatGPT search discovery. A publisher can decide which of these automated uses to permit rather than treating all OpenAI requests as one activity. (OpenAI crawler documentation ↗)
Jump to a section
Making one crawler decision at a time
A publisher disallows GPTBot because of its stated model-improvement use while leaving the separate OAI-SearchBot rule as configured. It records that choice by crawler and rechecks the live file; a rule for one crawler does not set the other crawler’s preference.
GPTBot is not the search crawler
Retain the rules for other crawlers when changing GPTBot’s group. A replacement robots.txt file that contains only the new block can accidentally remove unrelated instructions. The linked guide shows the separate groups and explains why infrastructure access also needs checking.
Treat search and training as separate settings
A publisher may want one crawler policy for training and another for discovery. Write down the intended result for each before editing the rules. Recheck both after deployment, especially when your platform replaces the whole file rather than updating one line.