Fetch as Bingbot, Googlebot or an AI crawler
Compare what your server returns to a normal browser and a selected search or AI crawler User-Agent. Find different status codes, redirects, titles, canonicals and robots directives.
Sends one ordinary browser request and one request with the selected crawler User-Agent. Requests come from this site's server, not from the crawler's verified IP range. No JavaScript rendering or robots.txt interpretation is performed.
See the exact User-Agent sent for Googlebot
Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)Test the response a crawler User-Agent receives
Enter the exact public page URL you want to diagnose, including its final path. Choose a crawler profile, then run the comparison. IframeTester makes two bounded requests from its server: one with an ordinary browser User-Agent and one with the selected bot User-Agent. It follows up to three validated public redirects per request and reads at most the first 48 KB of an HTML response. The tool returns a small diagnostic result, not the destination page body.
Compare the HTTP status and final URL first. A browser request that reaches the article while a bot-labelled request redirects to a login page or returns 403 is a useful lead. Then compare the page title, general and crawler-specific meta robots tags, X-Robots-Tag and canonical URL. Those fields can reveal User-Agent-dependent output even when both requests return 200.
The exact User-Agent string is visible beneath the form. These are documented example strings or published bot tokens; providers can change version numbers. The selector never accepts an arbitrary User-Agent, and the test never supplies a provider's verified IP address.
Search bots, AI search bots and on-demand fetchers
Search crawlers: fetch as Googlebot, Googlebot Smartphone, Bingbot, DuckDuckBot or Applebot when checking whether your public HTML differs by search crawler identity. A mobile Googlebot profile may also surface mobile-specific redirects or metadata that a desktop request misses.
AI search crawlers: test OAI-SearchBot, PerplexityBot, Claude-SearchBot or DuckAssistBot when investigating whether an AI search service's declared User-Agent gets a different response. These profiles are distinct from training crawlers in the providers' published documentation.
On-demand fetchers: ChatGPT-User, Perplexity-User and Claude-User identify visits associated with user requests. Use these profiles for a reported failure to open or cite a particular page. They do not all follow the same robots.txt behavior as automatic crawlers; this tool does not interpret those policies.
Training crawlers: GPTBot and ClaudeBot are separate profiles. Test them when you need to inspect HTTP behavior for those declared agents, but do not infer permission, training use or policy compliance from a 200 response alone.
What to check when a bot response looks wrong
Browser 200, crawler 403 or challenge
Look for a WAF, CDN bot rule, origin security plugin or rate limit that reacts to the User-Agent. Check its event logs for the same time and URL. Do not allowlist this tool's IP as if it were Bingbot, Googlebot or PerplexityBot; a real provider's identity requires its own published IP ranges or verification method.
Both requests return 200, but metadata changes
Compare titles, canonical links and robots directives. A bot-specific noindex tag or a canonical pointing somewhere unexpected deserves investigation in the server-generated HTML. If the page builds these tags only after JavaScript runs, this raw-HTML check may not see them. Use the relevant search engine's rendered-page inspection for the next step.
Both responses match, but a platform still cannot fetch the page
A matching comparison means only that these two requests from IframeTester's network returned the same inspected fields. The platform may use a different IP, location, HTTP behavior or rendering process. Robots.txt rules, authentication, TLS failures, blocked resources and a site's bot verification can all affect the real fetch. Check your logs and use Google Search Console URL Inspection or Bing Webmaster Tools where applicable.
Official crawler references
The profile names and example User-Agent strings come from these provider resources. Check them again before changing firewall rules because identifiers and IP ranges can change.
Common questions
How do I fetch a URL as Bingbot?
Paste the public page URL, choose Bingbot under Search crawlers, and select Compare responses. The tool requests the page once with a browser User-Agent and once with a Bingbot User-Agent, then shows differences in status, final URL and metadata. It does not send the request from Bing's network.
How do I test a page as Googlebot?
Choose Googlebot or Googlebot Smartphone and compare the response with the ordinary browser request. Check whether both reach the same final URL and return the intended title, canonical and robots directives. Use Search Console URL Inspection for Google's actual crawl and rendered view.
Can I test PerplexityBot and DuckDuckBot?
Yes. Choose PerplexityBot, Perplexity-User, DuckDuckBot or DuckAssistBot from the grouped selector. Search crawlers and user-triggered fetchers serve different purposes, so test the profile relevant to the problem you are investigating.
Why does a bot User-Agent get a 403 while my browser gets a 200?
The destination or its CDN may have bot-specific rules, rate limits or a challenge for requests that claim a crawler identity. The source IP is still IframeTester's, so a real verified crawler may receive a different response. Check firewall events and origin logs before changing an allowlist.
Does a matching 200 response prove a page is crawlable or indexed?
No. This tool does not interpret robots.txt, render JavaScript, follow internal links or make indexing decisions. It inspects a bounded slice of the raw HTTP response. A matching 200 is one useful signal, not an indexing verdict.
Does changing the User-Agent spoof the crawler's IP?
No. A User-Agent is a request header, while the source IP belongs to the network making the request. Services can verify known bot IP ranges or reverse DNS. This tester cannot bypass IP-based rules or CAPTCHA challenges.
What is the difference between an AI search bot and a training bot?
Providers publish separate identifiers for different uses. For example, OpenAI documents OAI-SearchBot for search and GPTBot for model training, while Perplexity documents PerplexityBot for search and Perplexity-User for user-initiated visits. A response comparison does not determine how a provider will use content.
Why are the page title or canonical missing?
The site may omit them from its initial HTML, return a non-HTML response, inject them with JavaScript, or place them beyond this tool's 48 KB read limit. Inspect the returned source and use the platform's official URL inspection tool to see any rendered result.