The file robots.txt can be inserted into your website’s root folder to help search engines to more appropriately index your site. Search engines such as Google use website crawlers or robots to check your website content. You may not wish to include parts of your site in user search results, such as the administrative page. The pages to be explicitly ignored can be added to the file. Robots.txt uses what the Robots Exclusion Protocol is called. With pages to be excluded, this site will easily generate the file for your website.
