Sitemap is a tool for site launch.
The process typically involves entering a web Address into a website based program, waiting for it to finish loading, downloading the Sitemap after the program is done working on the site, then uploading the Sitemap to the site's beginning directory without verifying the content in it.
This method leaves a lot of structural issues with the XML sitemap.
Today’s crawlers expect to see a high degree of accuracy when they crawl through web pages.
If an XML Sitemap is poorly created, it has the potential to waste crawl budget and send mixed signals to crawlers as to which web pages are the site owner’s intended primary pages of content.
To adequately evaluate a Sitemap creation Tool, Site owners need to consider the technical accuracy, scalability and crawling transparency of the Sitemap generation Tool, rather than getting caught up with online marketing campaigns that promote "instant Sitemap creation".
Why sitemap generation is important at this time
The current SEO landscape is filled with many thin reviews on low quality Sitemap generation Tools that merely repeat what is said on vendor websites.
The way that most companies currently are identifying the usefulness of these Tools has been through metrics that are frequently disregarded.
Sitemaps do not boost your search ranking.
A Sitemap is an aid to help search engines find and index your Web site.
When obtaining an accurate XML Sitemap, it is easy for search engines to quickly find your site's structure after a migration, a major update of content, or a redesign.
The distinction between a mediocre Sitemap generation Tool and an enterprise level Sitemap generator is based entirely on the differences in how the two handle Edge Cases; canonical tags, pagination, hreflang attributes and dynamic URLs.
Our research has demonstrated that the use of default setting of Basic Sitemap generation Tools can lead to including dead links and mismanagement of critical Indexing Tags.
The truth about the Sitemap Generator By Spellmistake
To get an idea of how a tool like the Sitemap Generator by Spellmistake works in practice, rather than relying on generic "How To" type articles and examples, we need to look at how well the tool produces crawls that can be reproduced.
There are so many different browser based tools available that produce a quick XML output.
However, the key to understanding the true value of a generator is to understand how it crawls a site.
Crawl speed and crawl completeness
The execution time and maximum URL limit are the first points of contention when crawling a site.
A lot of the free, browser-based tools will either time out when they are performing a crawl or they will have a maximum URL limit set at 500.
This is fine for a small brochure-type site.
But for an eCommerce site that has faceted navigation, having a maximum limit of 500 URLs is about as useful as having no limit at all.
A competent generator will be able to navigate deep into a site's structure without timing out and be able to distinguish what are valuable product pages from duplicate content that has been generated through the use of parameters.
How efficiently a website can be crawled directly impacts how useful the XML file that is created will be.
The importance of the lastmod tag
If there is one technical truth about SEO today, it is that the only sitemap signal that search engines consistently care about is the Tag.
In the past, SEO practitioners were very concerned about the Tag and the Tag.
But now the SEO industry and the documentation provided by search engines, specifically Google, place more emphasis on the Tag than they do on the Tag.
Searching algorithms depend heavily on a properly formatted Tag to determine if a page has been updated and needs to be crawled.
Many Sitemap Generators are creating a duplicate date for each URL in the sitemap; this is a bad practice.
A quality Sitemap Generator must take the real last modified date of a page from either the server or the CMS.
If the Generator is using a duplicate date, the search engines will become suspicious very quickly and the Sitemap will no longer be helpful to find fresh content.
How to test your Sitemap Generator (and why it is necessary)
A major gap in the current Sitemap Generation ecosystem is that most Sitemap Generators do not have any sort of method for Users to test the accuracy of their Sitemap.
Users are simply instructed to generate and upload a Sitemap, but they are not given the tools needed to test the output.
Modern web crawling issues like Automated Crawlers and Pagination
Most Sitemap Generators are still using scraping scripts written a long time ago, so they do not work correctly with the vast majority of modern Web Technologies.
Generating a sitemap with out paginated archives
Most Sitemap Generators built with an old scraping script will not follow a paginated Archive appropriately.
They will either end up in an infinite loop or miss some articles that are deep-linked.
Hreflang configurations
There is also a common issue with Hreflang Configurations.
International Websites that have complicated Hreflang Attributes must have a proper Hreflang configuration for the Sitemap, otherwise there will not be a proper URL match between the Hreflang configuration and the canonical URL listed in the header of the Page.
This will create a conflict in Google Search Console for indexing.
JavaScript
Client-side rendering can create a major challenge for Sitemap Generators; if the Generator does not have the ability to execute JavaScript during the scraping process, it will only see raw HTML Content.
For websites built using React, Angular, and Vue, without Server-side rendering, a basic crawler will only see an empty page with 0 Internal Links.
The sitemap you create will likely be mostly empty.
The privacy blindspot
Privacy is a conversation that is often overlooked regarding SEO Tools; However, it is one of the most significant vulnerabilities in this space.
When using cloud-based sitemap generator tools, you have allowed a third-party server to systematically scrape your entire site.
That raises the question of whether or not they store that information.
Are the URLs that Google encountered during the crawl stored on the third-party's server logs? Could any unlinked or staging environments have been unintentionally exposed to them?
Tools that do not have a specific data-retention policy poses a significant risk to you and your business.
Using offline or command-line interface (CLI) generators eliminates this risk because the entire crawl takes place locally.
Concrete verification workflow: Post-generation
Once you have uploaded your newly created XML file to the root directory of your site, that is not the end of your process.
That is just the start of your diagnostic process. You must verify that the sitemap is, in fact, impacting how the crawlers behave.
Search console log analysis
Once you have submitted the newly generated XML file to Google Search Console, you will need to monitor the status of the submission in real-time.
To check the status of the sitemap, you can go to the Sitemaps Report.
Initially, the status may say "Success"; however, this does not mean that the Sitemap is functioning as expected. The initial status only tells you that the XML syntax is correct; What you need to pay closest attention to is the number of "discovered URLs".
This number should match the number of canonically known pages within your website.
For instance, if the sitemap generator produced 10,000 URLs but Google Search Console has only indexed 4,000, this indicates a disconnect between the sitemap generator and the URLs that Google is able to crawl.
Most often, this is a result of the sitemap generator including URLs identified in robots.txt as blocked from crawling, or with a noindex directive tagged to the URL, which Google will disregard during a crawl.
Server-side crawl evidence
If you are looking for more certainty around the data produced by the sitemap, you will need to perform an analysis of the server logs.
To gauge the specific impact that the newly implemented sitemap has on your traffic, check your server log traffic for any Googlebot requests made to the sitemap URL before and after submitting it.
After these visits have occurred, begin tracking the number of times these deeper, previously "orphaned" URLs are crawled following the implementation of the sitemap.
If a correct format was utilized when extracting the data from the generator, you should see an increase in the amount of times crawlers have visited your recently modified pages.
This is the only way to determine definitively if your sitemap has functioned as designed.
Testing the limits of your XML file structure
Your strategy for creating sitemaps should be based on how scalable they are to accommodate a larger number of pages.
When developing a website at its beginning stages, a single sitemap that contains all of your pages is acceptable.
As the site continues to expand, a single large sitemap may not be practical due to the increased resource demand on server space.
Maximum file size/maximum number of URLs
Generally, Google does not permit sitemaps to exceed the following criteria:
- The total number of URLs contained within a single sitemap cannot exceed 50,000.
- The total size of an uncompressed XML sitemap cannot exceed 50MB.
When a site approaches the above limits, a sitemap index file must be created. An index file acts as a directory that tells search engine crawlers the location of multiple smaller sitemaps.
Effective diagnostic tool
Even if a site is significantly below the maximum number of URLs in a sitemap, segmenting sitemaps based on their content types allows for more effective diagnostic testing for indexing problems.
Rather than creating one large sitemap for a site, configure a sitemap generator to create multiple small sitemaps.
For example, create separate XML files for each component of the site; one sitemap for the blog, another for the products and another for the categories.
Each of these small sitemaps can be submitted to Search Console individually, which provides more directed insight into the indexing problems experienced in each section of the site.
When you use the sitemap generator, if the Product Sitemap has a 90% Index Rate and the Blog Sitemap is at 40% Index Rate, you know immediately that you should put your technical SEO focus on the Blog Sitemap.
If your sitemap generator is set to create one large, unordered list of every URL, you lose this ability to identify this critical issue with your indexation.
How to fix common XML errors post-generation
Even with the most sophisticated Crawlers, your generated XML Files can still trigger errors in Search Console and understanding how to interpret and fix these errors is important for maintaining crawling efficiency.
Fixing canonical conflicts
The most common error type indicates "Duplicate submitted URL not selected as canonical."
This indicates that your sitemap generator has generated a URL that points or redirects to another canonical URL.
When search engines see duplicate URLs, they will only index the true canonical target URL.
To resolve this issue, you need to change the crawler settings to adhere strictly to Canonical Tags so that only the final destination URLs are exported to the XML file.
Fixing robots.txt blocks
If Search Console tells you that the "Submitted URL Blocked by Robots.txt" error exists, this means that the sitemap generator did not parse your exclusion rules correctly.
Most basic tools will not follow exclusion rules and aggressively crawl your site ignoring Disallow directives and including Internal Search Result Pages, Login Portals, or Admin Directories in the sitemap.
This means you will need to manually clean up the XML prior to uploading the file or use a sitemap generator that natively reads robots.txt during the initial crawling.
The final word on creating sitemaps automatically
The number of available tools for creating sitemaps leans heavily towards ease of use rather than accuracy.
For small, simple websites (or static websites), being able to easily and quickly export a browser-based sitemap is sufficient.
For larger or more complex websites (or dynamic websites), it would be foolish to think that creating a sitemap is just an afterthought.
You need precision with your sitemap.
The best way to accurately create a sitemap is to use a tool that extracts the correct information from your website while also taking into account canonical tags and robots.txt directives, uses client-side rendering, and clearly defines the crawl paths for index segmentation.
A serious website administrator should also only use tools that are up-front about what data they keep on file (e.g., cached versions of the sitemap), as evidence-based SEO is all about what you send to search engines.
An accurately created sitemap is the foundation for having control over the signals you send to search engines.
Frequently asked questions
1) Will Google index my pages if I submit a sitemap to them?
No. A sitemap is a way for search engines to find your content, it does not guarantee that they will index it.
A sitemap provides a guide for crawling to find where to look for your content, but every URL must still be evaluated for its technical quality, uniqueness, etc. Before a search engine decides if it will include that URL in its index.
Submitting a sitemap will help the pages you submit be found more quickly and processed through the evaluation queue.
2) Should I be concerned about the priority and changefrequency tags in my sitemap?
Industry consensus and the search engines agree that the priority and changefrequency tags are now obsolete.
A decade ago, the tags had a purpose, but the current algorithms have matured enough to be able to identify how often each URL should be crawled based upon how the site has behaved historically and how much link equity it has received.
Therefore, any time you spend worrying about a specific priority and changefrequency score is time wasted.
Instead, you should focus entirely on ensuring your lastmod timestamps are as accurate as possible.
3) What should I do if I receive a robots.txt blocked error after creating my sitemap?
A robots.txt blocked error means that your sitemap contains URLs that your server has told search engines not to crawl.
You need to open your robots.txt file and check which directories are disallowed. You will then need to configure your sitemap generator to omit those paths.
If your sitemap tool does not allow you to exclude isolated paths, you will need to manually delete these URLs from the XML file before you submit it to Search Console.
4) Can a third-party website generator steal data from my website?
This is a valid concern.
Cloud-based sitemap generation tools crawl your website, generating the sitemap XML file on their own servers (the generator processes the HTML of your website).
If the generator does not clearly state how it handles privacy and data retention, there is the possibility that it could retain your website structure, any scraped content, or staging URLs.
If you are concerned about your privacy, use desktop-based crawl software or a Command Line Interface (CLI) generator that processes your entire sitemap locally on your machine.