Setting up PerplexityBot in your robots.txt file means adding the correct user-agent directive and crawl rules that tell Perplexity's web crawler which pages it can access and index. PerplexityBot is the crawler that Perplexity AI uses to build its search index and retrieve sources when generating answers. If your robots.txt blocks PerplexityBot, your content cannot appear in Perplexity citations regardless of how well it is optimized. DataReportal's Digital 2026 report confirms Perplexity attracts over 12 million unique monthly visitors, a discovery surface that requires explicit crawler access configuration.
What Is the Exact PerplexityBot User-Agent String?
The user-agent string for Perplexity's web crawler is PerplexityBot. Some older documentation references a variation, but the canonical and current user-agent is a single word. Add this directive to your robots.txt file.
User-agent: PerplexityBot Allow: /
If your robots.txt uses a catch-all Disallow directive followed by specific Allows, the order matters. Place the PerplexityBot Allow directive after any catch-all Disallow to ensure it takes priority. The following configuration correctly allows PerplexityBot while maintaining a restrictive default policy.
User-agent: * Disallow: /admin/ Disallow: /api/ Allow: /
User-agent: PerplexityBot Allow: /
The Princeton Generative Engine Optimization study found that technical crawl access issues are the single largest preventable cause of zero AI visibility. Brands that allow all major AI crawlers in their robots.txt see a measurable increase in citation frequency within 30 days of correcting access issues.
How Do You Verify PerplexityBot Can Access Your Site?
Verification takes two steps. First, check your robots.txt file by visiting yourdomain.com/robots.txt in a browser. Verify the PerplexityBot section exists and contains the correct Allow directives.
Second, check your server access logs for PerplexityBot crawl events. Search your logs for the user-agent string PerplexityBot. If you see crawl events after adding the robots.txt directive, Perplexity is successfully accessing your site. If you see no PerplexityBot entries within 14 days of configuration, check whether your CDN, WAF, or hosting provider is blocking the crawler at a different layer than robots.txt.
Some hosting providers and security plugins block known crawler user-agents by default. If you run Cloudflare, check your firewall rules and WAF settings for any rules that block bot user-agents. Add an exception for PerplexityBot if needed.
What Crawl Rate Should You Expect From PerplexityBot?
PerplexityBot crawls less aggressively than Googlebot or Bingbot. Expect crawl events on your site 1-3 times per week for established domains, and less frequently for new or low-authority domains. Unlike Google's continuous crawling, Perplexity crawls pages on a more deliberate schedule driven by query demand.
A page that receives a PerplexityBot crawl is likely being considered for citation within the following 24-48 hours. If you publish a new page and see a PerplexityBot crawl within 48 hours, consider it a strong leading indicator that your content is in Perplexity's retrieval pipeline.
If you are not seeing PerplexityBot crawl events, submit your sitemap URL through Perplexity's sitemap submission endpoint if available, or ensure your sitemap is discoverable from your robots.txt file with a Sitemap directive pointing to your full sitemap URL.