Nice RankMarketing
Request invite

AI Visibility Hub

AI Crawlers: search bots vs training bots in robots.txt

Robots.txt is a request, not a lock. Blocking training bots such as GPTBot and Google-Extended does not remove a site from AI search. Blocking search bots such as OAI-SearchBot and PerplexityBot does affect whether those search bots can access the site.

Nice Rank checks robots.txt daily and alerts when a search bot is blocked. Use that alert to start a clear decision about the public site. Do not treat a robots.txt change as a universal way to control every use of web content.

AI visibility work starts with the facts your business can support. Use the hub to spot places where a service, answer or page could be clearer, then review the result with the people who know the business best. Readiness checks and drafts are useful working tools, not a guarantee about how an AI search experience will represent the business.

How to approach it

  1. 01

    Review the daily check

    Open AI Crawlers to see the current robots.txt context and any alert.

  2. 02

    Tell training and search bots apart

    Read the bot type before changing a rule or drawing a conclusion about AI search.

  3. 03

    Resolve a blocked search bot

    Coordinate with the person who manages the website when a search bot needs access.