Robots.txt
August 02, 2014 | | Kris Tabetando | 0 Comment
Robots.txt is a robots text file which tells a search engine’s spider which web pages not to add to the search engine’s index.
Spiders are required to comply with the Robots Exclusion Standard. When they visit a website, they request a robots.txt file to learn what pages the spider is not allowed to index.
Author: Kris Tabetando
Kris Tabetando provides mergers & acquisitions (M&A) advisory and brokerage services to Internet companies. He also partners with investors to acquire & manage Internet businesses.
- Connect with Kris:
