DEVELOPMENT ENVIRONMENT

~liljamo/robots.txt

ref: 35b09eb73a85cf8bcf95249fea0ff5e9065a0ae8 robots.txt/lists/ai.txt -rw-r--r-- 336 bytes
35b09eb7Jonni Liljamo feat: add second nix package for src.quest robots.txt 28 days ago
                                                                                
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
360Spider

AI2Bot
Applebot
Applebot-Extended
AspiegelBot
Bytespider
CCBot
ChatGPT-User
Claude-Web
ClaudeBot
Diffbot
FacebookBot
GPTBot
Google-Extended
GoogleOther
ImagesiftBot
Kangaroo Bot
Meta-ExternalFetcher
OAI-SearchBot
Omigili
OmigiliBot
PerplexityBot
PerplexityBot
PetalBot
Timpibot
Webzio-Extended
YouBot

anthropic-ai
cohere-ai