# blopus.ai — robots.txt
#
# Blopus sells a search API to AI agents, so LLM crawlers are DELIBERATELY welcome:
# the comparison and pricing pages are how we get cited. No GPTBot/ClaudeBot/PerplexityBot
# blocks here, on purpose.
#
# 🔴 Disallow blocks CRAWLING, not INDEXING. A blocked URL still gets indexed when it is
# linked from anywhere, and Google cannot see a `noindex` tag on a page it is forbidden to
# fetch — which is exactly how /app ended up in the index. Pages we want OUT of the index
# are therefore CRAWLABLE and carry `<meta name="robots" content="noindex,follow">`.
# Only paths that must never be fetched at all are Disallowed.

User-agent: *

# Product surfaces: crawlable so their noindex is honoured, never indexed.
# (/app, /login, /support all carry a noindex meta tag.)

# Must never be fetched: internal, authenticated, or machine-to-machine.
Disallow: /admin
Disallow: /portal/
Disallow: /internal/
Disallow: /v1/
Disallow: /mcp
Disallow: /healthz

# Crawl-delay removed 2026-08-19: Google ignores it entirely and Bing honoured it,
# throttling the one crawler that was reading us for no benefit on a site this size.

Sitemap: https://blopus.ai/sitemap.xml