robots.txt & Sitemap Inspector

Fetches a site's /robots.txt, follows any Sitemap: directives it declares (or falls back to the conventional /sitemap.xml if none are declared), and validates both — crawl rules grouped by user-agent, syntax issues in robots.txt, and for each sitemap: URL count, priority/last-modified/change-frequency values, malformed-XML detection, and warnings for exceeding the sitemaps.org protocol limits (50,000 URLs / 50MB uncompressed per file). Sitemap index files are followed one level deep (first 3 child sitemaps). Browsers can fetch these files directly, but not read them cross-origin from a script the way this tool needs to — requires a live network request, made server-side. Nothing you check is logged or stored.

A note on scope

Syntax checking here is structural (unrecognized directive lines, Disallow/Allow with no preceding User-agent, unbalanced XML tags) — it doesn't second-guess your actual crawl policy choices (e.g. disallowing everything is valid, even if unusual). Sitemap parsing extracts the standard fields (loc, lastmod, changefreq, priority) from <urlset> or <sitemapindex> documents; sitemap index files are followed one level (a capped number of child sitemaps), not recursively without limit, and only the first 25 URLs of each sitemap are shown as a sample — the total count is still exact. See also the HTTP Header Inspector if you want the raw response headers for either file, and the URL Inspector for pulling apart any individual URL found inside a sitemap.