DEVELOPMENT ENVIRONMENT

~liljamo/robots.txt

ref: 602cb33d41088c238ed9102bd44a0510497d3460 robots.txt/lists/other.txt -rw-r--r-- 166 bytes
602cb33dJonni Liljamo docs: update README.md 28 days ago
                                                                                
1
2
3
4
5
6
7
8
9
10
11
12
13
# Apparently doesn't respect *
Seekport Crawler

# Alexa, what is my current location?
Amazonbot
meta-externalagent

# Meh
turnitinbot
Turnitin

# 80legs crawler
008