Robots.txt
August 02, 2014 | | Kris Tabetando | 0 Comment
Robots.txt is a robots text file which tells a search engine’s spider which web pages not to add to the search engine’s index.
Spiders are required to comply with the Robots Exclusion Standard. When they visit a website, they request a robots.txt file to learn what pages the spider is not allowed to index.
« Back to Glossary Index