CrawlTrack: Catch When Search Engines Visit Your Website

study search engine bot logs CrawlTrack: Catch When Search Engines Visit Your WebsiteToday we will share with you a dirty little secret you might not be aware of:  Most search engine robots use automated browsers (this includes spiders like Googlebot, Yahoo! Slurp, MSNBot.) When they visit your site, they will never execute the JavaScript in your source code to announce their visits. So, while Google Analytics (or scripts of that kind), do a wonderful job of tracking  human visitors, they cannot give you any information on spider (or User-Agent) behavior. Truthfully, to measure, or make the most of your SEO efforts, you need to know when a spider is coming in, what pages they’re requesting, and how often there coming back (if ever). With this in mind, we suggest you try out  the free robot tracker, Crawl Track. You can download it here. It is updated with all the major User-Agents, including Cuil and is activley being developped.  Not only do you get a detailed view of the crawler activity on your site(s), but you will also be able to track any bot-hacking attempts that are made (and have them blocked).  Other information you will have access to include: the bounce rate, the number of 404 errors, the number of file downloads, your server load usage and your site indexation. This is really a unique script that fills out your Web Analytics tracking. The best part is though, is the easy set up. All you need is a host with php (version >=4.3.0 with GD2) and one MySQL database. The installation is fully automatic. Remember, if you are not using Crawl Track or any Bot Log Analyzer, you will never know how often (and when) Google, Yahoo!, MSN  bots visit your site.

Print This Post Print This Post
  • Aug 19th, 2010 at 00:34 | #1

    I am using a website that I don’t design. Although I can change some basic stuff and add meta tags I can’t make an example/crawltrack page on site. Can I still use this? My site is http://www.BarrysHut.com

    Thank You.

  • Mark8t Staff
    Aug 19th, 2010 at 06:43 | #2

    No. You need access to page to enter info in the footer.

Leave a comment

XHTML: You can use these tags: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>