August 02, 2014  |     |     |   0 Comment

Robots.txt is a robots text file which tells a search engine’s spider which web pages not to add to the search engine’s index.

Spiders are required to comply with the Robots Exclusion Standard. When they visit a website, they request a robots.txt file to learn what pages the spider is not allowed to index.


Kris Tabetando provides mergers & acquisitions (M&A) advisory and brokerage services to Internet companies. He also partners with investors to acquire & manage Internet businesses.

    Connect with Kris:
  • linkedin
« Back to Glossary Index

Comments are closed.