What Is an XML Sitemap? Getting Found by Google, Bing and AI Crawlers
Key takeaways
|
Nobody gets excited about XML sitemaps. They're the plumbing of SEO.
But plumbing matters when the pipes are blocked. A sitemap stuffed with redirects, tag pages and URLs you've noindexed sends mixed signals and wastes crawl attention. A missing one can leave new content waiting weeks to be discovered.
That matters more now than it used to. AI assistants that search the web before answering pull from search indexes. If a page isn't discovered and indexed, it can't be retrieved, and it can't be cited.
What is an XML sitemap?
An XML sitemap is a file, usually located at yoursite.com/sitemap.xml, that lists the URLs you want search engines to crawl and index, often with the date each page was last modified. It helps Google, Bing and other crawlers discover new and updated pages faster, especially on large sites or for pages with few internal links. |
Where sitemaps came from and what they do
The sitemap protocol was agreed jointly by Google, Yahoo and Microsoft in 2006, and it's documented at sitemaps.org. It's one of the few things every major search engine still supports in the same way.
A sitemap doesn't force anything to be indexed. It's a list of suggestions: these are the pages I care about, and here's when they changed. Search engines still decide what's worth crawling and indexing. But a clean sitemap makes their job easier, and that makes your new and updated pages show up sooner.
What Google actually uses
Field | What it is | Does Google use it? |
loc | The page URL | Yes, it's the core of the file |
lastmod | When the page last changed | Yes, if the dates are consistently accurate |
priority | Your guess at a page's importance | No, Google ignores it |
changefreq | How often you think a page changes | No, Google ignores it |
The lesson: get lastmod right and stop fiddling with priority. If every page shows today's date on every crawl, search engines learn to ignore your dates entirely. Only update lastmod when the content meaningfully changes.
One more change worth knowing: Google retired its sitemap "ping" endpoint in 2023. If an old plugin or script still pings Google, it isn't doing anything. Submit through Search Console and reference your sitemap in robots.txt instead.
The Four Cs of sitemap hygiene
I audit sitemaps against four questions.
Canonical. Every URL is the canonical version. No duplicates, no parameter variations, no URLs that point their canonical tag somewhere else. See alternate page with proper canonical tag.
Current. Every URL returns a 200. No redirects, no 404s, no retired pages. See not found 404 and 301 redirects.
Complete. Every page you want found is in there, including new posts, service pages and key resources.
Clean. Nothing you've told search engines not to index. No noindexed pages, thin tag archives or internal search results.
Most CMSs, including Wix, generate a sitemap automatically. That's convenient, not a guarantee. Open the file and check what's actually listed.
How to submit a sitemap
Google Search Console
Add the sitemap URL in the Sitemaps report and check its status. Errors here are worth fixing first. Google Search Console for SEO covers the setup.
Bing Webmaster Tools
Submit the same sitemap to Bing. It's easy to skip, and it matters more than it used to because Bing's index feeds several AI experiences. See AI search visibility starts with Bing Webmaster Tools and Bing Webmaster Tools.
IndexNow
IndexNow, launched by Microsoft Bing and Yandex in 2021, lets you notify participating search engines the moment a page is added or changed. Many platforms support it natively. See what is IndexNow.
robots.txt
List your sitemap URL in robots.txt so any crawler can find it. Robots.txt for AI crawlers shows the format.
Why sitemaps matter for AI visibility
AI assistants that search the web, the process covered in search grounding, generally retrieve from search indexes. A page that hasn't been discovered and indexed can't be retrieved, so it can't be cited, no matter how good it is.
A clean sitemap speeds up discovery in Google and Bing, which feeds those indexes. Some AI crawlers also read sitemaps directly when they visit. See AI crawlers. It's one of the cheapest, least glamorous things you can do for AI visibility.
Common sitemap problems and fixes
Problem | What it looks like | Fix |
Submitted but not indexed | Search Console shows pages excluded | See crawled or discovered but not indexed guides |
Redirects listed | Sitemap URLs return 301s | Replace with final URLs |
Junk URLs | Tag pages, filters, internal search results | Exclude them from the sitemap |
Stale lastmod | Every page has the same date | Only update when content changes |
New posts missing | Fresh content not in the file | Check CMS settings, then request indexing for priority pages |
For the specific Search Console statuses, see crawled currently not indexed, discovered currently not indexed, URL Inspection and request indexing. The bigger checklist is in how to get indexed by Google and Bing indexing.
XML sitemaps in the Engine Optimization Matrix™
Sitemaps sit in the Distribution column of the SEO row in the Engine Optimization Matrix™: getting your content discovered and crawled. Everything in the GEO and LLMO rows that depends on retrieval starts here.
Related reading: what is search engine optimization, server-side rendering for AI crawlers, CMS for AI visibility and what is RAG.
Most teams still run SEO like it's 2019 and wonder why the traffic chart and the pipeline chart stopped agreeing. The Demand Doctrine™ for AI SEO covers what still works, what's dead and what to do Monday morning. Out November 18. Want all four engines? Mastering AI Visibility.
Frequently asked questions
What is an XML sitemap?
An XML sitemap is a file that lists the URLs you want search engines to crawl and index, often with each page's last modified date.
Where is my XML sitemap?
Usually at yoursite.com/sitemap.xml. Many CMSs, including Wix and WordPress with SEO plugins, generate it automatically.
How many URLs can a sitemap have?
Up to 50,000 URLs or 50 MB uncompressed per file. Larger sites use a sitemap index that points to several sitemaps.
Does Google use priority and changefreq?
No. Google has said it ignores both fields. It uses lastmod when the dates are consistently accurate.
Can I still ping Google with my sitemap?
No. Google retired its sitemap ping endpoint in 2023. Submit through Search Console and list your sitemap in robots.txt instead.
Should I submit my sitemap to Bing?
Yes. Bing's index feeds several AI experiences, so submitting to Bing Webmaster Tools helps both search and AI visibility.
What is IndexNow?
IndexNow is a protocol launched by Microsoft Bing and Yandex in 2021 that lets you notify participating search engines immediately when pages change.
Does a sitemap guarantee indexing?
No. It helps search engines discover pages, but they still decide whether each page is worth indexing.
About the author. Harold Bell is the founder of MQL Magnet and has spent about 16 years in enterprise technology marketing, working with brands like Cisco, Rubrik and Nutanix in different capacities. He's a member of the Forbes Communications Council, the creator of the Engine Optimization Matrix™ and the author of Partner Over Product. |
