An XML sitemap is a file that lists the URLs on your site you want search engines to crawl and index, along with signals like when each page last changed. It does not guarantee indexing, and it is not a ranking factor. What it does is give Google a clean, authoritative map of your important pages so they get discovered and refreshed faster, which matters enormously on large sites and barely at all on tiny ones.
Most site owners treat XML sitemaps as a box to tick and never look at them again. That is a mistake. A neglected sitemap is one of the most common sources of wasted crawl budget and confused indexing signals I see in audits, and it is one of the fastest technical fixes when it is wrong.
This guide covers what a sitemap actually does, the best practices that matter, when you need a sitemap index file, how to submit and monitor it, and the errors that quietly hurt you.
What an XML Sitemap Actually Does
An XML sitemap is a discovery mechanism. When Googlebot reads your sitemap.xml, it gets a list of candidate URLs plus optional metadata for each one, chiefly the lastmod timestamp. Google uses that list as one input into which pages it crawls and how often.
The critical thing to understand is what a sitemap does not do. Listing a URL in your sitemap does not force Google to index it. Google still decides indexing based on quality, duplication, and demand. According to Google’s sitemap documentation, a sitemap helps Google discover URLs but “doesn’t guarantee that all the items in your sitemap will be crawled and indexed.”
So the value is concentrated in two scenarios. First, large sites where internal linking cannot surface every URL efficiently. Second, sites with content that changes often, where the lastmod signal helps Google prioritise recrawls. On a well-linked 200-page brochure site, the sitemap is close to redundant.
There are also specialised sitemaps: image sitemaps, video sitemaps, and news sitemaps. Most sites do not need these. An image sitemap can help Google discover images loaded via JavaScript or served from a separate host, but if your images are in standard <img> tags, Google finds them through the page.
XML Sitemap Best Practices That Matter
Ignore the folklore. A handful of best practices carry almost all the weight, and the rest is noise.
Include only canonical, indexable, 200-status URLs. This is the single most important rule. Every URL in your sitemap should be a page you want indexed, that returns a 200 status, that is not blocked by robots.txt, and that is the canonical version. A sitemap full of redirects, 404s, noindex pages, or non-canonical duplicates sends Google contradictory signals and burns crawl requests on dead ends.
Keep lastmod honest. The lastmod tag should reflect the real last meaningful change to a page. Google has been explicit that it only trusts lastmod when the values are consistently and verifiably accurate. In 2023, Gary Illyes of Google confirmed that if a site sets every lastmod to the current date on every crawl, Google learns to ignore the field entirely for that site. A reliable lastmod on a large news or ecommerce site is a genuine crawl-efficiency lever; a fake one is worthless.
Respect the size limits. Google’s documentation caps a single sitemap file at 50,000 URLs and 50MB uncompressed. If you exceed either limit, split into multiple sitemaps and reference them from a sitemap index file. You can gzip sitemaps to reduce transfer size, and the 50,000-URL cap still applies to the uncompressed content.
Drop priority and changefreq. Google has said for years that it ignores the <priority> and <changefreq> fields. They are legacy hints that no longer influence crawling. Leaving them in does no harm, but do not spend a second tuning them.
Reference the sitemap in robots.txt. Add a Sitemap: https://example.com/sitemap.xml line to your robots.txt. This lets every compliant crawler, not just Google, discover it without manual submission. It is a one-line change that many sites forget.
If your sitemaps are automatically generated by your CMS, audit them anyway. Auto-generation is exactly where non-canonical and noindex URLs sneak in. When I run a technical SEO audit, the sitemap is one of the first files I diff against the live index, because the gap between the two tells a story fast.
When You Need a Sitemap Index File
A sitemap index file is a sitemap of sitemaps. Instead of pointing Google at one enormous file, you point it at an index that lists several child sitemaps, each covering a segment of your site.
You need one the moment you cross 50,000 URLs, but the smarter reason to use one earlier is diagnostics. If you split your sitemaps by section, product category, or content type, Search Console reports indexing coverage per sitemap. That segmentation turns a vague “some pages are not indexed” into “our blog is 98% indexed but our product pages are at 61%,” which is an actionable finding.
Here is a practical segmentation for a mid-size ecommerce site:
sitemap-index.xmlreferences all childrensitemap-products.xmlfor product detail pagessitemap-categories.xmlfor category and listing pagessitemap-blog.xmlfor editorial contentsitemap-static.xmlfor pages like About and Contact
Each child stays well under the 50,000-URL limit, and you gain a clean per-segment view of crawl and index health. This kind of structure pairs naturally with crawl budget optimization on large sites, because it lets you see exactly where discovery is breaking down.
How to Submit and Monitor Your XML Sitemap
Submitting an XML sitemap takes about a minute. Monitoring it is where the actual value lives.
Open Google Search Console, go to the Sitemaps report under Indexing, enter your sitemap path, and submit. Google fetches it, reports discovered URL counts, and flags parsing errors. You do not need to resubmit after each content change; Google recrawls known sitemaps on its own schedule.
The monitoring loop is the part most people skip. Watch three things:
- Discovered vs. submitted URLs. A large gap means your sitemap contains URLs Google refuses to count, often because they are non-canonical or blocked.
- The Page Indexing report cross-referenced with the sitemap. Filter indexing coverage by sitemap to see which segments underperform.
- Fetch status and errors. A sitemap that returns a 404, times out, or contains malformed XML silently stops working, and Google will not always shout about it.
Treat the sitemap as a live diagnostic instrument, not a submit-and-forget file. In my experience auditing sites, the sitemap-to-index gap is often the single fastest signal of a deeper technical or content-quality problem, and it costs nothing to check weekly.
The 7-Point XML Sitemap Audit
Run this checklist on any site you take over. It takes fifteen minutes and catches most sitemap problems before they cost you.
- Fetch the sitemap directly. Load
sitemap.xmlin a browser. Confirm it returns 200 and valid XML, not a 404 or an HTML error page. - Check the URL count against Search Console’s discovered count. A wide gap is your first red flag.
- Spot-check statuses. Crawl the sitemap URLs with a tool like Screaming Frog and filter for anything not returning 200.
- Look for
noindexand canonical mismatches. Every listed URL should be self-canonical and indexable. - Confirm robots.txt does not block listed URLs. A sitemap URL disallowed in robots.txt is a direct contradiction.
- Verify the robots.txt
Sitemap:line points to the correct, current sitemap path. - Validate
lastmodaccuracy on a sample of pages. Timestamps should match real edits, not the crawl date.
This process is deliberately mechanical. Sitemap problems are rarely subtle once you look; the failure is almost always that nobody looked. If you want this run across an entire site with the findings prioritised by impact, that is core to how I scope an SEO audit.
Common XML Sitemap Errors
These are the recurring failures I find, roughly in order of how often they appear.
Including non-canonical URLs. The sitemap lists ?utm_source variants, trailing-slash duplicates, or parameter URLs that all canonicalise elsewhere. Google has to reconcile the conflict, and you have diluted your own signal.
Listing redirects and 404s. Stale sitemaps that never got cleaned up point at pages that moved or died. Every one is a wasted crawl and a small credibility hit to the whole file.
Blocking the sitemap or its URLs in robots.txt. A surprisingly common self-inflicted wound, especially after a migration. If robots.txt disallows the paths your sitemap lists, you are telling Google to find pages it is not allowed to fetch.
Faking lastmod. Regenerating the file so every URL shows today’s date. As covered above, Google detects this and stops trusting the field, so you lose a real crawl-prioritisation signal.
Orphaned sitemaps after a redesign. The old sitemap path still returns content while a new one exists, and Search Console keeps reading the wrong file. Always confirm which sitemap Google is actually consuming.
None of these are complex to fix. They persist because sitemaps are invisible until someone audits them, which is exactly why they belong on a recurring technical checklist rather than a one-time setup task. Getting the sitemap right will not win you rankings by itself, but a broken XML sitemap can quietly cap how much of your site Google ever sees. If you would rather hand this off, this is the kind of unglamorous, high-leverage work a freelance SEO consultant should own end to end.
Frequently Asked Questions
Do I need an XML sitemap if my site is small?
Not strictly. Google crawls a small, well-linked site without one because internal links surface every URL. But there is no downside and one clear upside: a clean, canonical list of the pages you want indexed. Most platforms generate it automatically, so I recommend every site have one.
What is the difference between an XML sitemap and an HTML sitemap?
An XML sitemap is for search engines: a machine-readable list of URLs with metadata. An HTML sitemap is a human-facing page that links to your key sections for navigation. They serve different audiences, and the XML version is the one that matters for crawling and indexing.
Can an XML sitemap contain URLs from another domain?
Generally no. A sitemap should only list URLs on the same host it is served from. The exception is cross-submission verified through Search Console, where you prove ownership of both properties. For nearly all sites, keep the sitemap on the domain whose pages it covers.
Why are my sitemap URLs “Discovered – currently not indexed”?
That status means Google knows the URL exists but has chosen not to crawl or index it yet, usually a signal of low perceived value, thin content, or crawl-budget limits rather than a sitemap fault. The sitemap did its job by surfacing the URL; the fix is content quality, internal linking, and demand, not the sitemap itself.
Should I gzip my XML sitemap?
You can, and it is worth it for very large files to reduce transfer size and load. Google fully supports gzipped sitemaps. Just remember the 50,000-URL and 50MB limits apply to the uncompressed content, so compression saves bandwidth, not headroom against those caps.
