Crawler Tester
Fetch as Bingbot, Googlebot or an AI crawler. Check robots.txt and compare server responses.
Test the response a crawler User-Agent receives
Enter the exact public page URL you want to diagnose, including its final path. Choose a crawler profile, then run the comparison. IframeTester first checks the origin's robots.txt rules for that profile, reading up to 64 KB. If the path is allowed, it makes two bounded page requests from its server: one with an ordinary browser User-Agent and one with the selected bot User-Agent. It follows up to three validated public redirects, checks the robots policy at each destination, and reads at most the first 48 KB of an HTML response. The tool returns a small diagnostic result, not the destination page body.
The outcome panel separates an explicit robots.txt disallow, an uncertain robots policy, HTTP denial, rate limiting, server errors and applicable noindex directives. It suggests possible causes rather than claiming to know which firewall rule fired. Compare the HTTP status and final URL first, then the page title, general and crawler-specific meta robots tags, X-Robots-Tag and canonical URL. Those fields can reveal User-Agent-dependent output even when both requests return 200.
The exact User-Agent string is visible beneath the form. These are documented example strings or published bot tokens; providers can change version numbers. The selector never accepts an arbitrary User-Agent, and the test never supplies a provider's verified IP address.
Search bots, AI search bots and on-demand fetchers
Search crawlers: Choose Googlebot, Googlebot Smartphone, Bingbot, DuckDuckBot or Applebot to compare a public page's HTTP status, redirects and metadata with a browser response. If a mobile page is failing, compare Googlebot Smartphone with the desktop profile for different final URLs, titles or robots directives.
AI search crawlers: Choose OAI-SearchBot, PerplexityBot, Claude-SearchBot or DuckAssistBot to check the matching robots.txt rules and compare the response to a browser request. A successful test fetch does not establish that an AI search service will discover or cite the page.
On-demand fetchers: Choose ChatGPT-User, Perplexity-User or Claude-User when someone reports that an assistant cannot open a specific URL. Compare status codes and redirects, then check whether authentication or bot protection is involved. IframeTester honors matching robots.txt rules for every profile; providers may handle user-triggered visits differently.
Training crawlers: Choose GPTBot or ClaudeBot to inspect the HTTP response for those declared User-Agents. A 200 response shows what this test request received; it does not establish training use, permission or a provider's policy decision.
What to check when a bot response looks wrong
Browser 200, crawler 403 or challenge
Look for a WAF, CDN bot rule, origin security plugin or rate limit that reacts to the User-Agent. Check its event logs for the same time and URL. Do not allowlist this tool's IP as if it were Bingbot, Googlebot or PerplexityBot; a real provider's identity requires its own published IP ranges or verification method.
Both requests return 200, but metadata changes
Compare titles, canonical links and robots directives. A bot-specific noindex tag or a canonical pointing somewhere unexpected deserves investigation in the server-generated HTML. If the page builds these tags only after JavaScript runs, this raw-HTML check may not see them. Use the relevant search engine's rendered-page inspection for the next step.
Both responses match, but a platform still cannot fetch the page
A matching comparison means only that these two requests from IframeTester's network returned the same inspected fields. The platform may use a different IP, location, HTTP behavior or rendering process. Authentication, TLS failures, blocked resources and a site's bot verification can all affect the real fetch. The robots.txt check is bounded and may differ from an individual provider's handling. Check your logs and use Google Search Console URL Inspection or Bing Webmaster Tools where applicable.
Official crawler references
The profile names and example User-Agent strings come from these provider resources. The robots policy check follows the core matching rules in RFC 9309. Check provider resources again before changing firewall rules because identifiers and IP ranges can change.
Fetch as Bingbot or Googlebot: what to compare
To test as Bingbot, select Bingbot and enter the page URL that Bing cannot access. Compare its HTTP status and final URL with the browser response. A browser 200 and Bingbot 403 gives you a specific request to investigate in your CDN or origin logs; it does not prove that Microsoft's verified crawler is blocked.
For a Googlebot test, compare the desktop and Smartphone profiles separately. Look for different redirects, missing titles, unexpected canonical URLs or crawler-specific robots tags. The tester reads initial HTML. It does not render JavaScript or replace Search Console URL Inspection.
Test ChatGPT, Perplexity and Claude access
Choose OAI-SearchBot, PerplexityBot or Claude-SearchBot to investigate search discovery. For a user-reported failure to open a link, select ChatGPT-User, Perplexity-User or Claude-User. GPTBot and ClaudeBot are training-crawler profiles, so their results answer a different access question. Allowing a training bot is not a requirement this tool can establish for appearing in AI search.
Robots.txt access is different from indexing permission
A robots.txt Disallow rule asks a cooperative crawler not to fetch a path. A noindex directive in a meta robots tag or X-Robots-Tag asks supporting crawlers not to index a response they can read. Blocking the fetch can prevent the crawler from seeing noindex. A canonical URL is another signal; it does not grant access or guarantee which URL a search engine selects.
If the result says robots access is unknown, fix the robots.txt response before drawing conclusions about the page. After changing rules, use Fresh check & refresh robots.txt. A green fetch result means this request succeeded, not that a search engine indexed the page or an AI service will cite it.
Need to diagnose an embedded page instead? Use the iframe tester for X-Frame-Options and CSP frame-ancestors. Crawler access and permission to display a page inside an iframe are separate policies.
Common questions
How do I fetch a URL as Bingbot?
Paste the public page URL, choose Bingbot under Search crawlers, and select Compare responses. The tool requests the page once with a browser User-Agent and once with a Bingbot User-Agent, then shows differences in status, final URL and metadata. It does not send the request from Bing's network.
How do I test a page as Googlebot?
Choose Googlebot or Googlebot Smartphone and compare the response with the ordinary browser request. Check whether both reach the same final URL and return the intended title, canonical and robots directives. Use Search Console URL Inspection for Google's actual crawl and rendered view.
Can I test PerplexityBot and DuckDuckBot?
Yes. Choose PerplexityBot, Perplexity-User, DuckDuckBot or DuckAssistBot from the grouped selector. Search crawlers and user-triggered fetchers serve different purposes, so test the profile relevant to the problem you are investigating.
Why does a bot User-Agent get a 403 while my browser gets a 200?
The destination or its CDN may have bot-specific rules, rate limits or a challenge for requests that claim a crawler identity. The source IP is still IframeTester's, so a real verified crawler may receive a different response. Check firewall events and origin logs before changing an allowlist.
Does a matching 200 response prove a page is crawlable or indexed?
No. The tool checks robots.txt for the chosen profile and reads a bounded slice of the HTTP response, including robots directives. It does not verify the provider's IP, render JavaScript, follow internal links or make an indexing decision. A matching 200 is one useful signal, not an indexing verdict.
Will the tester fetch a URL disallowed by robots.txt?
No. It checks robots.txt for the selected crawler first and skips the page request when a matching Disallow rule applies. If robots.txt cannot be read reliably, it reports an inconclusive policy check and stops. A 404 or 410 robots.txt response is treated as no rules. The policy check is a bounded interpretation of public rules, not a guarantee of how each provider handles them.
Does changing the User-Agent spoof the crawler's IP?
No. A User-Agent is a request header, while the source IP belongs to the network making the request. Services can verify known bot IP ranges or reverse DNS. This tester cannot bypass IP-based rules or CAPTCHA challenges.
What is the difference between an AI search bot and a training bot?
Providers publish separate identifiers for different uses. For example, OpenAI documents OAI-SearchBot for search and GPTBot for model training, while Perplexity documents PerplexityBot for search and Perplexity-User for user-initiated visits. A response comparison does not determine how a provider will use content.
Why are the page title or canonical missing?
The site may omit them from its initial HTML, return a non-HTML response, inject them with JavaScript, or place them beyond this tool's 48 KB read limit. Inspect the returned source and use the platform's official URL inspection tool to see any rendered result.