15 lines
295 B
Plaintext
15 lines
295 B
Plaintext
# Added due to slurping up everything as fast as it could
|
|
User-agent: sistrix
|
|
Disallow: /
|
|
|
|
# I watched this bot for 4 consecutive days, it indexed the whole site 4 times
|
|
# each day (using GETs)
|
|
User-agent: AhrefsBot
|
|
Disallow: /
|
|
|
|
# Repeated indexing
|
|
User-agent: Ezooms
|
|
Disallow: /
|
|
|
|
User-Agent: *
|