Robots.txt

Google has open-sourced its ‘Web Crawler’ after two decades

If you are a web developer or publisher, you would have heard about robots.txt. For the people who are unaware, Robots.txt is Google's Robot Exclusion