Robots.txt & Sitemap Analysis

robots.txt and sitemap.xml guide search engines, but they also affect security and discoverability in ways people overlook. VitalSite checks that both exist and are healthy — and, importantly, that your robots.txt isn't quietly handing attackers a list of your private paths.

What VitalSite checks

• robots.txt accessibility. Whether the file exists and is reachable.
• Sensitive Disallow entries. Disallow lines that name admin, login, wp-admin, backup, config, or .env paths. Attackers read robots.txt too — disallowing a secret path doesn't hide it, it advertises it.
• sitemap.xml availability. Whether a valid sitemap exists, and how many URLs it lists.

Why it matters

Two opposite problems show up here. First, a missing sitemap hurts SEO and crawl coverage — search engines may not find all your pages. Second, an over-sharing robots.txt is a security smell: people routinely add Disallow: /secret-admin/ thinking it hides the page, when in fact robots.txt is public and the entry simply tells curious attackers exactly where to look. Blocking crawlers is not access control.

How to fix it

Add a healthy robots.txt that allows crawling and points to your sitemap:

User-agent: *
Allow: /
Sitemap: https://example.com/sitemap.xml

Don't rely on Disallow for security. Protect sensitive areas with real authentication, not robots rules. If a path must stay private, secure it — and then you don't need to mention it in robots.txt at all.

Generate a sitemap automatically (most CMSs and frameworks have a plugin or built-in feature) and submit it to Google Search Console so every page is discovered.

Verify the fix

Re-scan the domain. The Robots & Sitemap check should confirm robots.txt is present without advertising sensitive paths, and that a valid sitemap.xml is available with your expected URL count.

Run a free full-domain scan to see whether this affects your site — every page, with an exact fix guide. Pro and Agency plans add PDF export and monitoring.