# robots.txt

This repository contains my very opinionated (and possibly overengineered)
robots.txt generation.

`lists/` contains lists of user agents to disallow at root:
 * .txt files, line by line listings of user agents
 * empty lines are ignored
 * lines starting with # are ignored

`bases/` contains site specific base robots.txt files:
 * in the robots.txt format
 * names represent the domains they're served at

`generate.sh` is a bash script for generating output robots.txt files:
 * arg $1 is a required path to the out file
 * arg $2 is an optional path to a base file