c
j
d
Share this page

WildShark SEO Spider Tool

2015 is going to be a big year for WildShark SEO with the launch of our own range of SEO software and tools. In January 2015 we took on our own in-house software engineer who has the task of creating a range of SEO, marketing and sales related software that we can use ourselves and offer other technical professionals to use also.

Introducing the WildShark SEO Spider Tool

Here at WildShark we use a vast array of third party SEO tools every day. You name it – we probably use or have used it.

We fully understand what on-page elements need to be checked in order to create a fully optimised website with no crawl errors. We have used this knowledge to create our own SEO Spider Tool which makes it a lot easier for SEO professionals to spider through a website and pick up on anything that needs fixing or changing.

A tool developed for Web Design and SEO professionals by SEO professionals

Our SEO spider tool has been developed for anyone who works within the digital marketing industry. If you are web designer, it is the perfect tool to help you during a website build to ensure it is crawl friendly before launch. If you are an SEO expert, you can analyse all the crawl data to ensure your site is perfectly optimised. The tool has so many uses, including helping you to generate web design leads, easily.

FREE download with no limitations

The WildShark SEO Spider Tool has been developed as a desktop tool to be used on any Windows Vista, Windows 7 or Windows 8 platform. There are no limitations or costs associated with our tool. You will not be asked to upgrade or part with one penny. The WildShark SEO Spider Tool will crawl sites of any size and allow you to view and download a complete advanced crawl report amongst many other features. Download now to see how easy our tool is to use.

Are you a Mac user?

Although our tool is only available on Windows at this time. If we receive lots of requests for us to release a free Mac version. We will get it developed for you. All you need to do is email your request to [email protected]

Full feature list

SEO Spider

We do not see the point in adding hundreds of features in a tool that just do not need to be included. Our SEO Spider Tool is designed to check the things that really matter and below is a list of things our tool checks and reports on.

Generic features:

  • Crawl speed slider – We understand that, not every website is hosted on a good server. If your website is hosted on a low quality or shared server, you are able to limit the speed our tool crawls your website. However, if you are hosted on a great dedicated server – boost the crawl speed up to the maximum.
  • Skip externals – The tool is set as default to ignore external URLs. This speeds up the time it takes to crawl a website; especially if you have thousands of external links. If you uncheck the ‘skip externals’ box, you will be able to analyse all links pointing outside your domain.
  • Crawl error overview panel – The tool includes an overview window that displays all your crawl errors, what these errors are, the number of errors and where you can locate them (by clicking on the specific error).
  • View crawl errors on sites set to “noindex” – This is a great feature for web design agencies who need to spider a website set to “noindex” hosted on a demo domain during the site build. This enables you to launch a new website knowing it is search engine friendly.
  • Compatible with any language – Our SEO spider is being used by web and SEO professionals from all over the World. No matter what language your website content is written in, our tool will crawl it as normal and display the correct character set.
  • Ability to change the default user agent – Here is a list of all user agent strings. Just copy and paste your required string into the ‘User agent’ field.
  • User interface – Ability to float and dock the various panels in and outside the tool. You have total control over our user interface (UI) to drag and drop the various panels to match your own unique working style.
  • Filter panel – You have the ability to filter or exclude any data. As an example; if you want to exclude a complete sub-folder – you can. We have also included an advanced filtering feature which uses Regular Expressions (Regex).
  • Search panel – You have the ability to simply search through the URL, Title, Meta description and Headings fields to easily locate what you are looking for.
  • Overview: For web designer’s who have limited knowledge of SEO we have included a simple to understand overview feature that shows you how many errors you have, what they are and where they can be located. Check out the Web Designer’s SEO Tool Kit for simple user instructions.

Internal URL’s:

This displays similar to a spreadsheet document which includes the following columns:

  • URL – The full website address of that page. Click on any URL to open that page up within your browser.
  • Title – The page title between <title></title>. Click on the title to view it within the ‘Details’ panel.
  • Indexing – Allows you to see if a page is set to “follow”, “nofollow” or “noindex”.
  • Page linked from – This displays the top level page this URL is linked from.
  • Meta description – Click on the description to view it within the ‘Details’ panel.
  • Meta keywords – If meta keywords are used (which we do not suggest) the tool will display a list of keywords.
  • H1, H2, H3, H4 and H5 – Click on any heading tag to view the title, word count, number of characters and if it a duplicate H tag. This will appear within the ‘Details’ panel.
  • Google Analytics – Checks that page to see if the Google Analytics code is used.
  • CDN used – This stands for ‘content delivery network’ helps you to optimise the load speed of your website. The most common (and in our opinion the best) CDN is Cloudflare. It’s also great from protecting your website against attacks.
  • Word count – Displays the number of words between your opening and closing body tags (<body></body>).
  • Duplicate headings – Displays the number of duplicate H tags you have on any given page.
  • Canonical – Displays the URL used within the rel canonical tag of that page.
  • Broken links – Displays the number of broken links within any given URL / page.
  • Missing ALT tags – The number of image ALT tags that are missing on each page.

External URL’s:

This feature is disabled by default to speed up the crawl time. To enable it you simple uncheck the ‘Skip Externals’ check box located at the top of the tool. Why have we included external URL’s in the WildShark SEO Spider Tool? The main reason is so we are able to see if an external link is set to “follow” or “nofollow”. It is just a personal choice if you wish to pass any of your page authority on to the external URL.

  • URL – The exact URL used within the external link. Double click on it to open the external page up within your browser.
  • Title – The page title of the external URL. Linking out to other websites is a good thing if the page you are linking to is related to the page you are linking from. By viewing what the external title is, you can quickly see if it is related in some way.
  • Linked from – This is the page on your website that the external URL is linked from.
  • Rel Do or No Follow? – Are you passing part of your page authority (or link juice as many still call it) on to the external URL?
  • Telephone URLs – It is common now to add a link into all the telephone numbers displayed on your website so browsers using a mobile device can click on your telephone number and immediately call. All telephone links will appear within the ‘External URLs’ section.
  • Email URLs – Any email addresses linked via ‘mailto’ will be included within the ‘External URLs’ section.

Redirects:

We check all the links on your internal pages to see if any of them are using a 301 redirect. If so, you can change the link location to point to the correct page directly instead of going through the redirect.

  • URL – This is the URL used within your link.
  • Redirects to – This is the URL / page that it redirect to. Double click to open the page within your browser.
  • Original page URL – The internal page where the redirected link is displayed on.
  • Title – The page title of the redirected page.

Broken links:

Having links on your website pointing to an internal page that no longer exists is just not great for the user experience. In an ideal crawl, this section will display empty. You can also check out our ‘how to find broken links‘ guide.

  • URL – The location URL that no longer exists.
  • Linked from – The page that the broken link resides on.

Images:

A perfectly optimised website should use ALT tags related to the image. Each size of each image should also be as small as possible without losing the quality. There are various image compression tools available.

  • URL – The direct URL path to the image.
  • ALT – The ALT tag (if any) used on that image.
  • Image size – The file size (in KB) of all images.

Rich snippets:

By using rich snippets (Structured data markup) Google can use the data to index and display your content better. You can also add rich snippets to your pages to ensure your listing stands out from the crowd. You can also use rich snippets to control how your content is displayed when shared on social media platforms. Our tool checks for the following rich snippets:

  • URL – The URL of the page being analysed.
  • OG locale – The Open Graph protocol of ‘locale’ states what language your website is written in. The default is ‘en_US’ but if you wish to target France and you create pages in French then you need to set the locale to ‘fr_FR’.
  • OG type – This is the type of page. If the URL is a blog post then the ‘Og type’ will be ‘article’ but if it is a standard website page, it will be ‘website’.
  • OG title – The title of your object as it should appear within the graph.
  • OG description – This is similar to your meta description where you describe the object.
  • OG updated time – The date and time the last time the article was updated.
  • OG URL – The canonical URL of your object that will be used as its permanent ID in the graph.
  • OG sitename – If your object is part of a larger website, this is the name which should be displayed for the complete site.
  • OG image – This is the direct URL of each image.
  • OG video – The direct URL of the video file.
  • OG audio – The direct URL to an audio file to accompany this object.
  • Article section – The blog category your article is published in.
  • Article published – The time and date the blog article was first published.
  • Article modified – The time and date the blog article was last modified.
  • Twitter card – As the name suggests, this is the rich snippet related to Twitter. The Twitter card rich snippets specifies which card you want Twitter to generate.
  • Twitter site – This is your Twitter username (@wildsharkseo).
  • Twitter creator – This is usually, the Twitter ID of the author of the article.
  • Twitter description – The description of the content you are promoting.
  • Twitter title – This is the title of your page you wish to use on the Twitter card (the tweet).
  • Twitter image source – The direct URL of the image you wish to use within the tweet.

Keyword search:

This is a great feature we use all the time in-house here at WildShark. The keyword search section allows us to perform analysis including:

  • Displays which page is most likely to rank for a certain keyword or phrase you input.
  • Displays all pages related to a certain phrase.
  • Allows you to see if your site is over optimised for a given phrase.
  • Add industry related phrases into the search bar to see if you have a related page on your website. If not, get one written to maximise your exposure.
  • Visualise how a page displays within the search engine listings. Being able to view how a page will look within the Google listings is great as you can ask yourself – If I was searching for ‘your phrase’ would this make me click on my website instead of one of my competitors?
  • Checks how many times that keyword appears within various elements (URL, title, meta description, meta keyword, H1, H2, H3, H4 and H5) of that page.
  • Double click on any listing to display that page within your browser.

Speed:

Having a fast loading website is so important these days. Everyone is impatient and just will not wait any length of time for a page to load. They will just click straight off and go to one of your competitors. You need to ensure that every page on your website loads within 5 seconds as it helps your rankings as well as the overall customer experience. Our tool looks at the following points regarding speed:

  • URL – The page URL being tested. Double click on it to open the page up within your default browser.
  • Response time – This is the time taken (in ms) of the request between the server and browser. Issues relating to your server / hosting.
  • Page download time – The time taken (in ms) to download the page after your browser has connected to your server. Issues related to your website.
  • CDN – If you are using a content delivery network to optimise the speed of your website, the tool will display which CDN you are using.
  • Size – This is the total size in KB of each page. We like to keep our pages under 500 KB but sometimes, this is just not possible.
  • Amount of images – The number of images on each page / URL.
  • Compression – Gzip compression can reduce the size of each page by around 70% by compressing the coding of each page. The tool will show ‘No’ if your pages are not compressed or ‘Yes’ if it does.

Export:

Many of the web and marketing agencies use our report export feature as a great sales tool to generate new customers. They first search for their target client. Run their website through our SEO spider tool. Download the error overview report using their own logo and send the report to their prospects. In most cases the prospect is shocked at the number of errors that exist on their website and ask for a price to fix them. There are other great features within our ‘Export’ section though:

  • Advanced report – The advanced report is basically a replica of what you see within the ‘Internal URLs’ tab. You have the ability to download the advanced report as either CSV, XLSX, XML or JSON.
  • Error overview report – This is a one page PDF report which you can add your own logo to. It displays the URL crawled, total number of crawl errors, total URL’s crawled, description of each error and the number of errors associated with each error type. each error type is colour matched according to how bad the site is. red being bad and green being good. Makes everything very simple to understand. Usually the agencies who use this tool create a before and after error overview report to send to their client.
  • Sitemap – This export feature allows you to generate and download an XML sitemap for both pages and images based on your crawl. You have the ability to include or remove certain settings from your XML sitemap.

Help / Support:

If you encounter and issues with the WildShark SEO Spider Tool or you would just like to request a new feature, all you need to do is click on the ‘help’ link within the tool and follow the instructions. As our software engineers are employed in-house, your issue will be fixed within 24 hours (normal business hours). If you are having an issue installing our tool, just email the details of your problem to [email protected]

Future plans for the WildShark SEO Spider Tool

Our SEO spider tool will always be free to use with no restrictions. We will continue to develop new features into the tool and over the next few weeks we will be putting together in-depth documentation and how-to guides to support our free SEO Spider Tool.

All that is left to say is – go ahead and download now to view the complete features. It’s free with no limitations.