Share
What is the Robot.txt?
For those of you who don’t know what robot.txt is; it is a protocol which is designed to limit what areas of sites can be crawled by spiders, to understand this completely you should have a rough idea of how spiders work. Using the robot.txt optimisation carries with it major benefits when used correctly. For example forums will regularly use this feature, i.e. in general chatter sections if they do not want the member’s general discussions to confuse a search engines designation of their theme. Think of it as a no follow tag for content, be it images, banners or entire pages.
How do I use robot.txt optimisation?
Well it is relatively simple piece of code, for example if you wanted to prevent the Googlebot from seeing graphics on your site the code you would place in the robot.txt file would be:
User-agent: Googlebot
Disallow: /graphics/
However most of this can be done using the Meta tags if for some reason you cannot upload a robot.txt files.
Another key use of the protocol is preventing any on site duplicate content penalties, for example if you have two pages with similar or identical content you can use the Robot.txt file to tell the spiders to ignore the content which you do not wish to be the authoritative piece. This is very helpful when you have a large site in a competitive or highly plagiarised field.
Limitations of the Robot.txt
The robot.txt file is quite limited for example a search engine may still index a file that it the robot.txt file is telling it not to crawl if another site is linking to that page. Also a more concerning issue is that the Robot.txt file only effects “well behaved” robots; it will not stop a competitor from analysing or attacking your site.
How do I create a robot.txt file?
Robot.txt files are pretty easy to make, not only are there numerous online tools to help you make them, or you can write it yourself in a text editor and upload it to the root directory.
Share With Microsoft desperately trying to move forward in its ever loosing battle to provide the number one search engine it has signed non exclusive deals with both Facebook and Twitter in an attempt to integrate there live feed update services to Bing. Is this a sign of desperation on Microsoft’s part? Trying to cash in on the traffic of social networking sites, or a stroke of genius? Only time will tell. But I doubt that this latest investment in the flailing, floundering search engine Bing will have the guys at Google quivering for too long, not only is the deal not exclusive there is no sign as of yet that it will be lucrative.
So why have Microsoft done this?
Well, to put it simply, Microsoft are failing to keep up with Google and even the lesser used Yahoo, while they may have had the dominance in the internet browser market which more than gently nudges you towards using Bing I might add they haven’t actually managed to convert this into any sort of useful marketing leap.
The other main reason is that they have to get involved in this. Google are in talks with Facebook and Twitter and of course Microsoft cannot afford to fall behind again especially now Google looks set to get another jump on them by indexing AJAX based sites. Google’s index is already staggeringly greater than Microsoft’s and more accurate, thus it works better for users so more people use it, simple really.
The corporate jugernaught that is Microsoft seeks to dominate all things in the computer world and is slipping on many fronts. With apple nipping at Microsoft’s heals and Google trouncing it in the online market they really need to pull something out of the bag and soon lord knows they have the money. So the real question is what the hell is it exactly that Microsoft are playing at? We all saw what happened to Netscape when they took Microsoft on; they were crushed completely and utterly. Though this move will undoubtedly be a bonus to Bing users the real question is, is it a bonus to social network users and is it enough of a bonus to convince people to use Bing. Well I would suggest it is quite unlikely even if they do manage to get it all integrated before Google.
Although social networks do absorb a great detail of our lives has it got to the point where they can be the tipping point in corporate competitions? I doubt it. Let’s just hope that the arrival of windows 7 is well received or the giants may be seeing some dark days in the near future.
