
Robots.txt for SEO Ranking
What is
"robots.txt"?
For those
who don’t know what is robots.txt, it is a text file which instructs search
engine robots to crawl a page or not. Any CMS (content management system) based
sites are having admin module online and that should not be crawled by Search
Engines, using robots.txt you can block that part to not get it crawl.
About /robots.txt
Web site owners use
the /robots.txt...