DeepCrawl (now Lumar) is a mean green crawling machine. Our crawling capabilities are unmatched in terms of scale, delivery and depth of reporting to name but a few.
Sadly, not everyone is aware of the benefits of DeepCrawl and cloud crawling solutions, as we’re often grouped in with desktop crawlers that simply can’t compete with us on a number of fronts.
Given these misconceptions, we thought it would be useful to highlight the key points that differentiate DeepCrawl from desktop crawling solutions to help you understand how our crawler better equips you to improve the technical health of the sites that you manage.
1. Scale – Why Size Matters
How is DeepCrawl different?
Local crawling tools are limited by your computer’s memory and will likely be limited to thousands of URLs per crawl. In contrast, cloud-based crawlers are distributed across many servers so you aren’t limited by the speed and size of your machine.
What are the benefits?
As a result of our innovative approach, DeepCrawl has no problem crawling small sites, incredibly large sites and everything in between.
Our crawler can comfortably cope with 10 million plus URLs in a single crawl, which is why we’re trusted by today’s behemoth sites and the majority of Fortune500 companies.
In 2017 alone, we crawled over 3 billion URLs and completed over 400,000 crawls!
2. Flexible Speed & Scheduling – The Convenient Crawler
How is DeepCrawl different?
Given that you don’t need to rely on your machine’s resources, that you have the freedom to crawl whenever and wherever is convenient.
No more dedicating a desktop machine to running crawls and keeping it on while it crawls a site. You can easily schedule regular crawls to run at a time and frequency of your choosing.
You can even set off a crawl from your mobile and see the results later on your desktop – useful for when you need to garner data for those last minute pitches.
What are the benefits?
With DeepCrawl, you are fully in control of your crawl:
- You choose the speed of your crawl, empowering you to gain insights quickly but without overloading your site’s servers
- You choose the time of your crawl, allowing you to avoid peak traffic times and putting unnecessary stress on your server
- You choose the frequency of your crawl, enabling you to regularly check for changes every day, week or month
3. Third-party Integrations – Boldly Going Where No Crawler Has Gone Before
How is DeepCrawl different?
DeepCrawl is about so much more than basic web crawls. We go further than any other crawler on the market to integrate your can’t-live-without SEO tools, to provide you with a holistic overview of a website’s technical health and performance.
When you set up a crawl we give you the option to bring in data from:
Backlink tools
Our partnership with Majestic means you can bring backlink data into your crawl at no extra cost and in just a couple of clicks (you don’t even need to have a Majestic subscription!).
Log file analysers
Integrating log file summary data from tools like Splunk shows you how Google crawls your site, and combined with crawl data, confirms whether Google is paying attention to the right pages, or not!
Organic Search Data
Hook up your Google Search Console account to throw search engine ranking performance data into the mix.
Web Analytics Data
Bring in data from Google Analytics as the final piece of the puzzle in understanding websites from how they are crawled by search engines to how visitors actually interact with them.
What are the benefits?
Layering these data sources together within our platform saves you time and effort that would have otherwise been wasted manually joining this data together in Excel spreadsheets.
The net result is that, with DeepCrawl, you can get to the core of the issues on a site much faster by bringing together many different data sources in one place.
4. API Access – Tap Into Our Back End
How is DeepCrawl different?
Our API provides programmatic access to our award-winning crawler, empowering you to utilise our crawler’s functionality outside of the standard DeepCrawl interface. Not to mention you can fully white-label our crawl data with your own logo and custom color scheme.
What are the benefits?
Using DeepCrawl’s API, you can write applications that make the most of DeepCrawl’s many features, deployed in your own environment, using the programming language and platforms that you prefer.
Using our API you can:
- Build custom dashboards
- Automatically trigger a crawl whenever a site is updated
- Integrate SEO data into client-facing platforms
5. Simultaneous Crawling – Crawls, On Crawls, On Crawls
How is DeepCrawl different?
With DeepCrawl, there’s no need to wait around for one crawl to finish before setting off another, our sophisticated crawling technology runs crawls, large and small, all at the same time. You can start as many crawls as you like and we’ll send you a notification when they’re ready for you, to analyse in neatly organised project folders.
What are the benefits?
DeepCrawl cuts down on the time it takes to set up crawls, freeing you up to spend the time on what matters – uncovering issues and insights! So you can make websites more visible in search engine result pages and provide better user experiences.
6. Scheduling – Crawl Like Clockwork
How is DeepCrawl different?
Setting up regular crawls with a desktop crawler is a cumbersome task which involves running scripts from the command line or by using cron jobs. DeepCrawl strips out this unnecessary layer of complexity and allows you to easily schedule crawls at regular intervals.
On top of that setting up a crawl is a simple 4 step process:
- Enter the domain you want to crawl
- Add in the sources from your other available SEO tools
- Set the speed, scale and depth of your crawl
- Specify any other settings dependent on the requirements of your crawl
Then hit ‘Save and Start Crawl’ – it’s as easy as that!
What are the benefits?
Scheduled crawling is a massively popular time-saving feature in our platform. It means you don’t have to remember to start a crawl every hour, day, week, or month, simply select your desired crawl frequency and wait to be notified when your crawl has finished AKA…set it and forget it!
An added benefit of scheduled crawling is that you will soon build a backlog of previous crawls that can be easily referenced throughout our reporting enabling you to spot changes from crawl to crawl, for trend data made easy!
7. Crawl History – Started From a Test Crawl Now We’re Here!
How is DeepCrawl different?
DeepCrawl organises crawls chronologically within projects, so you always have the option of going back and looking through previous crawls.
We visualise your most recent crawls in trend graphs throughout our reporting so you can monitor how you’re tracking on any metric from crawl-to-crawl.
What are the benefits?
With DeepCrawl, we save you the effort of having to organise and maintain folders of historical crawl data. All of your crawls are saved within projects for you or your team members to access forever and always.
Trend graphs are also a massive time-saver, that let quickly spot changes between crawls and easily communicate your successes with your boss or clients or give the dev team a kick up the ass to fix issues as they arise.
8. Communicate Your Insights – Sharing is Caring
How is DeepCrawl different?
Multiple Users
The beauty of cloud-based crawling, is that data isn’t tied to any one individual user. Accounts can have multiple users attached to them so that different team members can run and access crawls without using a shared login.
Sharing Reports
If you want to share your crawl insights without giving out access to your account, you can do so with a simple share link or by sending the report in an email. As our crawl graphs/reports are downloadable as PDFs, PNGs, XLSX, you can easily share as much of our datasets as you like.
DeepCrawl can even be whitelabeled and configured with your or your clients’ logo and brand colors.
Task Management
Equally, if you’ve identified an issue that you’d like to assign to someone else you can make use of our Task Management tool which allows you to allocate tasks via email as well as setting deadlines and severity flags. This makes it easy to see how long various SEO activities take you (or your teams) to complete.
It’s worth noting that all ‘Tasks’ are highlighted in their own report, so you can essentially customise your own report within DeepCrawl, where all of the SEO issues that matter to you most can be monitored in one place – at a glance!
What are the benefits?
Whether you’re making use of multiple users, share links, branded reports, task management or all three, sharing crawl insights is always a breeze.
The bottom line is that your crawl data isn’t held by the one person running the crawls. Insights can be easily shared and tasks easily assigned to others within SEO and development teams but you can also share our easy-to-interpret visualisations with clients and management teams.
But Don’t Take Our Word For it…
If you need to crawl many large sites simultaneously on a regular basis while also needing to easily layer data sources to uncover and communicate clear and actionable insights, look no further than DeepCrawl!
The above points are just some of the reasons why DeepCrawl is such a powerful companion for search marketers, but don’t take our word for it. Try out DeepCrawl for yourself. We have a flexible range of packages tailored to suit your needs.
