An XML sitemap lists your URLs for search engines, but its mere presence does not guarantee the crawling or indexing of your pages. The real question is what the crawl reports reveal once the file is submitted: is the issue related to the discovery of the URLs, their perceived quality, or a technical blockage that the sitemap will not resolve?
Search Console Diagnosis: sitemap, discovery, and indexing are not the same issue
When Google Search Console reports non-indexed URLs, three categories of causes coexist. Confusing them leads to modifying the sitemap when the problem lies elsewhere.
| Type of problem | Symptom in Search Console | Can the sitemap help? |
|---|---|---|
| Discovery | URL missing from coverage report, status “Detected, not yet crawled” | Yes: adding to the sitemap signals the URL to the crawler |
| Crawling blocked | “Blocked by robots.txt” or server error (5xx) | No: the sitemap does not bypass a technical blockage |
| Quality / relevance | “Crawled, not indexed” or “Duplicate page without canonical URL” | No: Google deems the content insufficient or redundant |
A URL marked “Detected, not yet crawled” means that Googlebot knows the address but has not yet visited it. The sitemap plays its role here: it accelerates discovery. On the other hand, in the case of the status “Crawled, currently not indexed,” the sitemap has fulfilled its mission, the problem lies elsewhere.
Segmenting your sitemap by content type (product pages, articles, categories) allows you to filter Search Console reports by file. You can then identify if an entire group of pages has a quality issue, without mixing signals.
Value of lastmod and thematic segmentation of the XML sitemap
The lastmod tag must reflect a real change in content, not a simple footer tweak or cosmetic date change. Google uses this value when it is consistent and verifiable. A sitemap where all dates change every day loses all credibility in the eyes of the crawler.
You will find additional resources on Connect Info’s website regarding the structure and management of sitemap files.
Thematic segmentation goes beyond the technical size constraint (the limit remains set by the sitemaps.org protocol). Creating a file dedicated to product sheets, another for editorial content, and a third for category pages offers a concrete advantage: in Search Console, each sitemap has its own coverage report.
- A product sitemap where the majority of URLs show “Crawled, not indexed” points to a problem of duplicate or thin content on the sheets
- An editorial sitemap with URLs “Detected, not yet crawled” indicates a lack of internal linking or insufficient crawl budget
- A category sitemap showing canonicalization errors reveals a conflict between parameterized URLs and clean URLs
XML Sitemap and IndexNow: two complementary mechanisms
The sitemap provides a lasting inventory of all URLs to be indexed. IndexNow serves to notify certain engines of a one-time change (addition, update, deletion). Bing recommends using both in conjunction: the sitemap for the overall mapping, IndexNow for real-time signals.
Notification via IndexNow does not guarantee indexing. It accelerates the consideration by Bing and Yandex crawlers. Google has not adopted the protocol to date, making the XML sitemap even more central to the Google ecosystem.
Sitemap extensions based on content type
Google documents specific extensions for images, videos, and hreflang annotations. Each extension addresses a distinct indexing need: the image extension allows signaling visuals hosted on a CDN different from the main domain, the video extension provides duration and thumbnail, and hreflang manages language targeting between versions of the same content.
Enabling all extensions on a site that only publishes text brings no benefit. The choice is made based on the content actually present:
- E-commerce site with photo galleries: image extension for each product sheet
- Media publishing video reports: video extension with complete metadata (title, description, duration, thumbnail)
- Multilingual site: hreflang annotations in the sitemap rather than in each head tag, to simplify maintenance
- Monolingual blog without video or gallery: no extension needed, the standard XML sitemap suffices
Internal linking and sitemap: what the XML file does not replace
A sitemap signals URLs to crawlers, but it does not transmit any authority between pages. Internal linking remains the lever that distributes link weight (internal PageRank) and prioritizes your content in Google’s eyes.
An orphan page added to the sitemap may be discovered, but rarely indexed sustainably if no internal link points to it. The status “Crawled, currently not indexed” in Search Console often concerns pages accessible only via the sitemap, without a navigation context.
Check consistency between sitemap and architecture
Regularly compare the URLs in your sitemap with those actually linked in your structure. Any URL present in the sitemap but absent from the internal linking is suspicious. Either it deserves a contextual link from a parent page, or it should not be in the sitemap.
This cross-checking, combined with segmented Search Console reports, transforms the sitemap into a diagnostic tool rather than a mere technical formality. The XML file does not fix anything by itself, but it makes visible what is malfunctioning in your architecture.
