Robot.txt Optimization
What is the Robot.txt?
For those of you who don’t know what robot.txt is; it is a protocol which is designed to limit what areas of sites can be crawled by spiders, to understand this completely you should have a rough idea of how spiders work. Using the robot.txt optimisation carries with it major benefits when used correctly. For example forums will regularly use this feature, i.e. in general chatter sections if they do not want the member’s general discussions to confuse a search engines designation of their theme. Think of it as a no follow tag for content, be it images, banners or entire pages.
How do I use robot.txt optimisation?
Well it is relatively simple piece of code, for example if you wanted to prevent the Googlebot from seeing graphics on your site the code you would place in the robot.txt file would be:
User-agent: Googlebot
Disallow: /graphics/
However most of this can be done using the Meta tags if for some reason you cannot upload a robot.txt files.
Another key use of the protocol is preventing any on site duplicate content penalties, for example if you have two pages with similar or identical content you can use the Robot.txt file to tell the spiders to ignore the content which you do not wish to be the authoritative piece. This is very helpful when you have a large site in a competitive or highly plagiarised field.
Limitations of the Robot.txt
The robot.txt file is quite limited for example a search engine may still index a file that it the robot.txt file is telling it not to crawl if another site is linking to that page. Also a more concerning issue is that the Robot.txt file only effects “well behaved” robots; it will not stop a competitor from analysing or attacking your site.
How do I create a robot.txt file?
Robot.txt files are pretty easy to make, not only are there numerous online tools to help you make them, or you can write it yourself in a text editor and upload it to the root directory.















Search engines are notorious for their inability to update quickly, the idea of the search engine spiders being able to crawl the entire web for new pages at a moments notice was simply unfathomable. The power involved in searching the entire internet and keeping the search engines as close to real time as we know them is incredibly vast and that is nothing in comparison to what updating the internet in real time would take.

Page rank is one of the main factors taken into account when Google decides what page to place your page on. So this makes having a good page rank very important, so if your page rank is a little to low, then this could be why you don’t rank very well and that is why you could need to increase page rank.



