#robots.txt

This repository contains my very opinionated (and possibly overengineered) robots.txt generation.

lists contains lists of user agents to disallow at root

• .txt files, line by line listings of user agents
• empty lines are ignored
• lines starting with # are ignored

bases contains site specific base robots.txt files

• base robots.txt files in the robots.txt format
• names represent the domains they're served at

generate.sh is a bash script for generating output robots.txt files

• arg $1 is a required path to the out file
• arg $2 is an optional path to a base file