Feature Update: Separate Mobile User Agent
With Google’s switch to mobile-first indexing, it is important than ever to have a complete view of how both the mobile and the desktop versions of your site are seen by users and search engines. You can achieve this by crawling the desktop and mobile versions of your site in the same crawl, but this can be tricky if you have a separate mobile site.
The DeepCrawl team are pleased to announce the launch of our mobile user agent which allows you to crawl desktop and mobile sites within the same crawl.
Crawl Desktop & Separate Mobile Websites Within the Same Crawl
Often separate mobile websites will redirect between the desktop and mobile sites depending on the user agent. Up until now DeepCrawl has only been able to run one user-agent per crawl, meaning some users wanting to crawl separate mobile websites have had to maintain separate projects with crawls using different user agents.
We understand the importance of maintaining a separate mobile site and its relationship to its desktop equivalent. As such, we’ve made some fundamental improvements to our crawler, that enable you to crawl a desktop and separate mobile site within the same crawl.
What are the key benefits of this update?
- If you have a separate mobile site which redirects based on the user agent, you can now crawl both versions of your site in the same crawl.
- You can now identify separate mobile pages within a crawl without relying on the presence of a rel alternate tag.
- You can test mobile and desktop redirects in a separate list-only crawl by applying alternate user agents for desktop and mobile pages.
How can I set this up?
Taking advantage of this handy update is super-simple and can be set up in a few easy steps.
1. Specify a second user-agent – In the fourth stage of the crawl setup, head to Advanced Settings > Spider Settings > Mobile Site and you can specify a second user agent which will be used for separate mobile pages and avoid the redirection.
2. Enter your mobile homepage – Entering a Mobile Homepage will ensure that the crawl starts on the mobile homepage in addition to any other start URLs specified in your crawl. If your mobile site is hosted on a different domain or subdomain, this will automatically be included as a secondary domain.
3. Enter your Mobile URL Pattern – The Mobile URL Pattern lets our crawler know which URLs will use the mobile user agent.
Although this latest update is only relevant if you are crawling sites with a separate mobile configuration, you can find information about crawling dynamically-served sites with different user agents in our in-depth product guide.
Gain a complete picture with DeepCrawl
If you’re concerned about how your site is shaping up for the mobile-first future, then we’re here to help. DeepCrawl’s best in-class crawl insights and extensive mobile reporting will put you on the fast-track to success in search. If you’re interested in benefiting from the power of an enterprise crawler, then why not drop us a message or get some first-hand experience with your own DeepCrawl account.
Already a DeepCrawl user? Then pat yourself on the back for making such sound life choices, before logging into DeepCrawl and setting up crawls with a second mobile user agent.
