# robot eXclusion principle
User-agent:	*
Disallow:	/stats/
# 13/9/04 - Stop "Slurp" slurping too often (not more than once every 5 minutes; later changed to one hour)
User-agent:	Slurp
Crawl-delay:	3600
# 14/2/05 - Ditto "msnbot" but with delay of one hour
User-agent:	msnbot
Crawl-delay:	3600
# 4/4/05 - Allow "googlebot" to once again roam these 'ere shores
User-agent:	googlebot
Crawl-delay:	3600
# 16/9/07 - The NLA's PANDORA robot seems to have tied itself in a knot
#	http://pandora.nla.gov.au/crawl.html
#	In any case, I don't know why this site should interest the NLA;
#	just because it's operated from Australia doesn't mean...
User-agent:	archive.org_bot
Disallow:	/
# 2/3/09 - Majestic-12 search engine is running amok; install temporary
#	block to identify genuine agents, i.e., those that respect this
#	command, or more generally "robots.txt", from the rest
User-agent:	MJ12bot
Disallow:	/