# What does robots.txt ask crawlers to do?

robots.txt is a public text file that asks compatible crawlers which website paths they may fetch.

Compatible crawlers read rules identified by user-agent names and website paths. These instructions communicate crawl preferences; they are not login credentials or a paywall.

## Choosing the right kind of restriction

Example: robots.txt disallows a compliant crawler from /members/. If /members/ remains a public URL, a person who enters that address can still receive the page; the crawler rule is not a password. The publisher uses authentication if the content must be private.

## Crawl rules are not access control

Blocking crawling is also different from asking a search engine not to index a URL. For example, a crawler blocked from fetching a page cannot read a noindex instruction on that page. Choose the control for the outcome you need: crawl access, search indexing or private access.

## Check the right control for the job

If you need to keep a draft confidential, test whether an unauthenticated reader can open it. A crawler instruction is a different control. If you are changing crawl access for public material, check the live file and the response at an example public URL.

## Related terms

[OAI-SearchBot](https://digitalpublishingnews.com/dictionary/oai-searchbot/)

[GPTBot](https://digitalpublishingnews.com/dictionary/gptbot/)

## Continue reading

[OAI-SearchBot and GPTBot: choose search access separately from training access](https://digitalpublishingnews.com/articles/openai-search-training-crawlers/): It explains the crawl-request convention and why it cannot secure a public URL.

## Sources

https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers

https://www.rfc-editor.org/rfc/rfc9309.html

https://developers.google.com/search/docs/crawling-indexing/robots/intro

Editorial method: https://digitalpublishingnews.com/how-we-use-ai/

Canonical: https://digitalpublishingnews.com/dictionary/robots-txt/