robots.txt + sitemap validator
Validate syntax + sitemap references — prerequisite for clean indexing.
How this tool works
robots.txt controls how search crawlers visit your site — Allow/Disallow rules per user agent, crawl delays and sitemap references. sitemap.xml helps crawlers efficiently find all your URLs. This validator loads both files live, parses the syntax (user-agent blocks with their Allow/Disallow rules, all mentioned sitemaps), follows sitemap references, and checks each sitemap.xml for valid XML, format (urlset vs sitemapindex) and URL count. Spot typos, missing sitemap references or broken XML files instantly.
Typical use cases
- After an SEO audit, verify nothing important is accidentally blocked from crawlers
- Before submitting to Google Search Console, ensure XML validity
- On indexing issues, check whether robots.txt accidentally has Disallow: /
- Verify the sitemap reference in robots.txt — without it, crawlers may not find the sitemap
Frequently asked questions
Which user agents should I explicitly name in robots.txt?
"User-agent: *" as default, plus optional "Googlebot" for Google-specific rules. Specific rules override "*". Not needed: individual bots like "Bingbot", "DuckDuckBot" — they usually use "*".
What is the difference between urlset and sitemapindex?
urlset contains your URLs directly (max 50,000 / 50 MB per file). sitemapindex references multiple urlsets — essential for large sites. You can reference just one sitemapindex in robots.txt and structure sub-sitemaps underneath.
Crawl-delay — when needed?
On small servers that get overloaded by crawlers. Google ignores it by the way — use Search Console for that. Bing, Yandex and Baidu honour crawl-delay.
My robots.txt has absolute URLs — is that OK?
Disallow expects relative paths (Disallow: /admin/), not absolute URLs. The sitemap reference however MUST be absolute (Sitemap: https://...).
How many sitemaps can I list?
Any number in robots.txt. We check the first 3 because more is rare and load times would explode.
Related tools
Continuous monitoring vs. one-shot?
XICTRAQ monitors uptime, SSL, security headers, GDPR and performance 24/7. Free plan, no credit card.