Crawler permission, search eligibility, citation and traffic are separate stages. A robots.txt rule can permit a supported crawler to fetch a public page; it does not guarantee indexing, an AI recommendation or a visit.
Separate search from training
OpenAI documents OAI-SearchBot for search and GPTBot for potential model-training use. Their controls are independent. Blocking GPTBot does not itself block OAI-SearchBot or make ChatGPT search citations impossible. User-triggered requests have a separate role. OpenAI crawler documentation.
For Google's AI Overviews and AI Mode in Search, normal Google Search eligibility and Googlebot controls apply. Google-Extended is a separate control; it is not the switch for appearing in those Search features. Google's AI-feature guidance.
Perplexity also distinguishes its search crawler from user-triggered fetching. Use its current documented agents and verification guidance rather than treating every bot as interchangeable. Perplexity crawler documentation.
An example of separate choices
A publisher may want search access while opting out of GPTBot crawling:
User-agent: OAI-SearchBot
Allow: /
User-agent: GPTBot
Disallow: /
This is an illustration, not a complete robots file for every business. Review the whole configuration before changing it. More-specific agent groups have their own matching behavior; repeat any needed private-path restrictions in the applicable group. Robots.txt is not authentication or protection for confidential material.
Check actual access
Inspect the public HTTP status, robots directives, indexing directives and canonical for representative pages. Keep the styles and scripts needed to render public content accessible. Check verified crawler traffic and responses at the CDN and origin; a user-agent string alone is not proof of identity.
A missing crawler visit within an arbitrary number of days does not prove a firewall block. The crawler may not have chosen to fetch the page. Look for actual blocked requests and use the platform's documented verification information before changing firewall rules.
Measure each stage independently
Track crawl access, indexed/retrievable pages, observed answer citations, referral visits and qualified demand separately. A correct robots file is a technical prerequisite in some workflows, not an AI-ranking tactic or a substitute for useful content. Training access is a separate publishing decision, not a required trade for search eligibility.
Conbersa's scope
This is educational guidance. Conbersa's current service is managed social account operations for customer-supplied content. It does not guarantee search citations or provide the SEO/GEO service implied by some older descriptions. Real-phone account operations also do not guarantee protection from platform restrictions.