Crawler · SiteCheckBot
SiteCheckBot
SiteCheck uses an automated crawler to inspect websites you submit or monitor. The crawler only fetches publicly accessible pages over HTTP or HTTPS.
User-Agent
SiteCheckBot/1.0 (+https://sitecheck.litework.me/bot)
What it does
- Fetches pages from URLs you enter or add to your dashboard.
- Follows internal links up to your plan's page limit.
- Reads robots.txt and sitemap.xml when present.
- Checks links, HTTP status, redirects, SEO tags and TLS.
Blocking SiteCheckBot
To disallow crawling, add rules to your site's robots.txt:
User-agent: SiteCheckBot Disallow: /
We honour robots.txt directives. If you block the bot, scans may return incomplete results or fail for that site.
Contact
Questions about crawling or abuse reports: support@litework.me.