CrawlTrack: Catch When Search Engines Visit Your Website
Today we will share with you a dirty little secret you might not be aware of: Most search engine robots use automated browsers (this includes spiders like Googlebot, Yahoo! Slurp, MSNBot.) When they visit your site, they will never execute the JavaScript in your source code to announce their visits. So, while Google Analytics (or scripts of that kind), do a wonderful job of tracking human visitors, they cannot give you any information on spider (or User-Agent) behavior. Truthfully, to measure, or make the most of your SEO efforts, you need to know when a spider is coming in, what pages they’re requesting, and how often there coming back (if ever). With this in mind, we suggest you try out the free robot tracker, Crawl Track. You can download it here. It is updated with all the major User-Agents, including Cuil and is activley being developped. Not only do you get a detailed view of the crawler activity on your site(s), but you will also be able to track any bot-hacking attempts that are made (and have them blocked). Other information you will have access to include: the bounce rate, the number of 404 errors, the number of file downloads, your server load usage and your site indexation. This is really a unique script that fills out your Web Analytics tracking. The best part is though, is the easy set up. All you need is a host with php (version >=4.3.0 with GD2) and one MySQL database. The installation is fully automatic. Remember, if you are not using Crawl Track or any Bot Log Analyzer, you will never know how often (and when) Google, Yahoo!, MSN bots visit your site.
Print This Post
