DEVELOPMENT ENVIRONMENT

~liljamo/robots.txt

ref: 35b09eb73a85cf8bcf95249fea0ff5e9065a0ae8 robots.txt/lists/other.txt -rw-r--r-- 166 bytes
35b09eb7 — Jonni Liljamo feat: add second nix package for src.quest robots.txt 28 days ago 
                                                                                
6a060977 Jonni Liljamo 













28 days ago













1
2
3
4
5
6
7
8
9
10
11
12
13

# Apparently doesn't respect *
Seekport Crawler

# Alexa, what is my current location?
Amazonbot
meta-externalagent

# Meh
turnitinbot
Turnitin

# 80legs crawler
008