1 files changed, 4 insertions(+), 4 deletions(-)
M README.md
M README.md => README.md +4 -4
@@ 3,16 3,16 @@
This repository contains my very opinionated (and possibly overengineered)
robots.txt generation.
-`lists` contains lists of user agents to disallow at root
+`lists/` contains lists of user agents to disallow at root:
* .txt files, line by line listings of user agents
* empty lines are ignored
* lines starting with # are ignored
-`bases` contains site specific base robots.txt files
- * base robots.txt files in the robots.txt format
+`bases/` contains site specific base robots.txt files:
+ * in the robots.txt format
* names represent the domains they're served at
-`generate.sh` is a bash script for generating output robots.txt files
+`generate.sh` is a bash script for generating output robots.txt files:
* arg $1 is a required path to the out file
* arg $2 is an optional path to a base file