Robots.txt is important on every website or blog because this file will be submit on Google Webmaster tool, this process instruct the google web crawlers on how to index and crawl your website in the search results. Robots.txt is a text which contains few lines of syntax for web crawlers. It is saved on the […]