Compatible crawlers read rules identified by user-agent names and website paths. These instructions communicate crawl preferences; they are not login credentials or a paywall. (Google common crawlers documentation ↗ · IETF RFC 9309: Robots Exclusion Protocol · Google robots.txt limitations and indexing)
Jump to a section
Choosing the right kind of restriction
Example: robots.txt disallows a compliant crawler from /members/. If /members/ remains a public URL, a person who enters that address can still receive the page; the crawler rule is not a password. The publisher uses authentication if the content must be private.
Crawl rules are not access control
Blocking crawling is also different from asking a search engine not to index a URL. For example, a crawler blocked from fetching a page cannot read a noindex instruction on that page. Choose the control for the outcome you need: crawl access, search indexing or private access.
Check the right control for the job
If you need to keep a draft confidential, test whether an unauthenticated reader can open it. A crawler instruction is a different control. If you are changing crawl access for public material, check the live file and the response at an example public URL.