#robots.txt

This repository contains my very opinionated (and possibly overengineered) robots.txt generation.

lists/ contains lists of user agents to disallow at root:

• .txt files, line by line listings of user agents
• empty lines are ignored
• lines starting with # are ignored

bases/ contains site specific base robots.txt files:

• in the robots.txt format
• names represent the domains they're served at

generate.sh is a bash script for generating output robots.txt files:

• arg $1 is a required path to the out file
• arg $2 is an optional path to a base file

generate-nginx.sh is a bash script for generating an nginx if block to block user agents at that level:

• prints out an if block with every user agent in the lists