Our dedicated DeepCrawl engineers are constantly building, testing and pushing SEO to its limits. So it seems only right to share their findings and help you fine-tune your strategy even further. Introducing our new series on SEO experiments: DeepCrawl Labs…
The test
Sitemaps can sometimes take significant amounts of time to generate, and we wanted to test how Google reacted to sitemaps with differing response times.
The method
We set up multiple XML sitemaps with varying response/load delays to see if Google would be patient enough to wait for them. These ranged from 100 to 350 seconds.
The results
Sitemaps with a load time of up to 2 minutes/120 seconds were generally accepted without issues. Nothing over 200 seconds was ever indexed.
Some slower sitemaps between 120-200 seconds showed a timeout error, but also showed that they were later accepted with submitted and indexed items.
What this means for you
We noticed that when crawling sitemaps with longer loading times, the crawl speed was negatively affected. We therefore suggest aiming for a 60 second load time for your sitemap to be sure they aren’t causing any problems.
We also recommend using lots of smaller sitemaps, rather than one big one, as this can reduce the sitemap response times.
If your sitemaps are still taking too long to generate, it may be possible to pre-generate them periodically and cache them on the server.
For tips on creating sitemaps, conducting audits or advanced configuration, have a read of this guide.
