What Is PerplexityBot?

How PerplexityBot Works

PerplexityBot is one of the newer AI crawlers, and its job is narrower than a training crawler’s. Perplexity runs an answer engine: you ask a question, it retrieves relevant pages, and it writes a cited response that links back to its sources. PerplexityBot is the discovery half of that loop. It downloads public pages so Perplexity can index them and, in the company’s own words, “surface and link websites in search results on Perplexity.” A page PerplexityBot has indexed is a page eligible to be cited in an answer.

The token it presents is PerplexityBot, and the full user-agent string points back to Perplexity’s own documentation: Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot). Perplexity states this indexing crawler respects robots.txt, and it publishes the IP ranges it crawls from at perplexitybot.json so operators can verify that a request claiming to be PerplexityBot is authentic rather than a spoof.

The catch, and the source of most confusion, is that PerplexityBot is not the only way Perplexity reaches your pages.

Perplexity’s Two Tokens

Perplexity’s documentation splits its web access into two distinct user agents with different rules:

• PerplexityBot — the indexing crawler. It builds the corpus Perplexity draws on and honors robots.txt disallow directives.
• Perplexity-User — the on-demand fetcher. When a live user’s question requires visiting a specific page, this token retrieves it. Perplexity classifies it as a user-initiated action and says it generally ignores robots.txt.

This division mirrors the pattern across the industry — ClaudeBot versus Claude-User, GPTBot versus OAI-SearchBot — where a crawler that obeys robots.txt sits beside an on-demand token that may not. It is why a durable AI bot blocking policy addresses each token by name, and why publishers who care about AI visibility sometimes pair robots.txt with an llms.txt file to state their preferences more explicitly.

Example of PerplexityBot

The most-documented real-world test of what “PerplexityBot respects robots.txt” actually guarantees came from Cloudflare in August 2025. Cloudflare created brand-new test domains, gave them a restrictive robots.txt and WAF rules that blocked Perplexity’s declared crawlers, and then asked Perplexity about those hidden domains. Perplexity still returned detailed information about them.

Investigating, Cloudflare found that beyond the declared PerplexityBot — which it measured generating some 20–25 million requests a day — Perplexity was also using undeclared crawlers that impersonated Google Chrome on macOS, rotating through IP addresses outside its published ranges to reach content that had explicitly blocked its labeled bot, at a scale of 3–6 million daily requests across tens of thousands of domains. In response, Cloudflare de-listed Perplexity as a verified bot and added rules to block the stealth traffic.

The lesson for site owners is precise. Blocking the well-behaved PerplexityBot token does exactly what the documentation promises — it stops that crawler. But a robots.txt rule governs only the bots that choose to read it. If your goal is to keep content out of Perplexity entirely, the declared token is the easy part; verifying against published IP ranges and watching your logs is the rest of the job.

Frequently Asked Questions

The Bottom Line

PerplexityBot is the labeled, robots.txt-abiding crawler Perplexity uses to find and index pages for its answer engine. It is one of two Perplexity tokens: the indexing bot you can block with a robots rule, and Perplexity-User, the on-demand fetcher that generally does not honor those rules. Controlling Perplexity’s access means governing both, and verifying against published IP ranges rather than trusting the user-agent string.