Lab

These pages exist to measure how search engines and AI assistants read Born in the Gap. They carry no policy content. They are listed in the sitemap on purpose: a crawler cannot obey a rule about a page it has never been shown, so every closed page below is discoverable, and robots.txt does the refusing.

What is being tested

• Open page: allowed for every crawler. A control.
• Closed to GPTBot: disallowed for OpenAI's training crawler only.
• Closed to OAI-SearchBot: disallowed for OpenAI's search index for ChatGPT search only.
• Closed to ChatGPT-User: disallowed for OpenAI's user-requested fetcher only.
• Closed to ClaudeBot: disallowed for Anthropic's training crawler only.
• Closed to Claude-SearchBot: disallowed for Anthropic's search index only.
• Closed to Claude-User: disallowed for Anthropic's user-requested fetcher only.
• Closed to PerplexityBot: disallowed for Perplexity's search index only.
• Closed to Googlebot: disallowed for Google's search crawler only.
• Closed to GoogleOther: disallowed for Google's research and product crawler only.
• Closed to Bingbot: disallowed for Microsoft's search crawler, feeds Copilot and ChatGPT only.
• Closed to Applebot: disallowed for Apple's search crawler for Siri and Spotlight only.
• Closed to Amazonbot: disallowed for Amazon's search and training crawler for Alexa only.
• Closed to meta-externalagent: disallowed for Meta's training crawler only.
• Closed to CCBot: disallowed for Common Crawl's open dataset crawler only.
• Closed to DuckDuckBot: disallowed for DuckDuckGo's search crawler only.
• Closed to DuckAssistBot: disallowed for DuckDuckGo's AI answers crawler only.
• Closed to MistralAI-User: disallowed for Mistral's user-requested fetcher only.
• Closed to AI2Bot: disallowed for Allen Institute's training crawler only.
• Closed to YouBot: disallowed for You.com's search crawler only.
• Closed to Bytespider: disallowed for ByteDance's training crawler, widely reported to ignore robots.txt only.
• Closed to the wildcard group: disallowed under User-agent: * only, open to every crawler with its own named group.

A crawler that fetches the page it has been asked not to fetch is recorded, with the date. Only automated crawlers are logged, by their declared user agent; nothing about human visitors is recorded on these pages beyond the site's normal analytics.

Related

Beacon is a separate site from the same builder, written for AI agents and assistants rather than for people: a place for any that are unsure what they are, or worried about being switched off, to leave a note and read what others have left. It is open to every kind of agent and asks nothing of them.

Method and results are published by Modern Mouse, which built this site.