About the Shelob Web crawler

User-agent: ShelobWeb/1.0 · Last updated: September 22, 2026

Identifying the crawler

Shelob Web is a website health monitoring service. When a customer adds a site to their dashboard, our crawler checks that site’s pages for SEO, accessibility, performance and AI-readiness issues, and reports them back to the site’s owner.

Our requests carry this user agent:

ShelobWeb/1.0 (+https://shelobweb.com/bot)

We only crawl sites that a signed-in customer has explicitly added to their account. If you are seeing this crawler on your site and you don’t know why, someone with access to your dashboard added it — or you can block it outright using the instructions below.

What it requests

• robots.txt first, to find your sitemap and to read your crawling rules.
• Your sitemap, and then only the URLs your sitemap advertises. We do not spider your link graph or explore parameterised URLs.
• Page HTML — nothing else. We do not download images, stylesheets, scripts, fonts or video.
• HEAD requests (no response body) for links that appear on your pages but not in your sitemap, to detect broken links.
• llms.txt and llms-full.txt, if present, for AI-readiness scoring.

We never attempt to access password-protected or private pages, never submit forms, and never log in.

Performance measurements do not touch your server at all: Core Web Vitals come from Google’s Chrome UX Report field data, and Lighthouse runs on Google’s PageSpeed infrastructure.

How often and how fast

• Once per day per site, overnight (around 3am US Pacific). Viewing a dashboard never triggers a crawl — results are read from storage.
• At most 500 pages per site per run.
• At most 10 requests in flight at a time, with a pause between batches.
• At most 100 HEAD requests per run for broken-link checking.
• When a site is first added, we check the homepage only — a single request.

Blocking or slowing it down

We honour robots.txt. Rules addressed to ShelobWeb take precedence over wildcard rules, and we support Disallow, Allow, and Crawl-delay.

Slow us down

A crawl delay drops us to one request at a time, spaced at your requested interval:

User-agent: ShelobWeb
Crawl-delay: 10

We never crawl faster than the delay you set. A long delay simply means we cover fewer of your pages each night — we pick up where the budget runs out on subsequent runs.

Keep us out of certain paths

User-agent: ShelobWeb
Disallow: /admin/
Disallow: /checkout/

Block us entirely

User-agent: ShelobWeb
Disallow: /

Changes take effect on the next nightly run, since we re-read robots.txt at the start of every crawl. Note that blocking us means the site’s owner will stop receiving health reports for it.

Questions

If this crawler is causing problems on your server, or you believe it is crawling a site without authorisation, email us and we will stop it from our side:

Email: support@shelobweb.com