Skip to main content
S
sitemoraβ

sitemora-bot

About the sitemora website structure analysis crawler.

User-Agent

Mozilla/5.0 (compatible; sitemora-bot/1.0; +https://sitemora.app/bot)

Our crawler sends every request with the User-Agent above. Any access containing "sitemora-bot" in your server logs is our crawler.

Crawler behavior

  • We crawl only URLs that users explicitly submit to sitemora. We do not perform automated or speculative crawling.
  • We fetch robots.txt per domain and honor Disallow directives. The number of URLs excluded is shown in the job results.
  • We honor Crawl-delay directives and space out requests accordingly (capped at 30 seconds).
  • On 429 / 503 / 504 responses we retry once, honoring the Retry-After header.
  • Concurrent connections per domain are capped at 10 (or 1 when Crawl-delay is set).

For site operators

To control sitemora-bot individually, add the following to your robots.txt.

Disallow entirely

User-agent: sitemora-bot
Disallow: /

Disallow specific paths

User-agent: sitemora-bot
Disallow: /admin/
Disallow: /private/

Request a crawl interval

User-agent: sitemora-bot
Crawl-delay: 5

Contact

For questions or requests about our crawler, please reach out via one of the following.