Commit Graph

2 Commits (1a10b64425d2eba6dcdd602a779bcd7b7d212195)

Author SHA1 Message Date
Cătălin Mariș a7326cc270 Update `robots.txt` file
* Update the URL to `www.robotstxt.org`.
* Add comment making it clear to everyone that the directives from
  the `robots.txt` file allow all content on the site to be crawled.
* Add `Disallow:` directive.

Ref: 4e5f438e1e
2014-06-20 00:02:43 +03:00
Addy Osmani 11f223c837 Introduce file renames. Minor restructuring. 2014-04-17 13:02:52 +01:00