DEVELOPMENT ENVIRONMENT

~liljamo/robots.txt

602cb33d41088c238ed9102bd44a0510497d3460 — Jonni Liljamo 27 days ago 6a06097
docs: update README.md
1 files changed, 4 insertions(+), 4 deletions(-)

M README.md
M README.md => README.md +4 -4
@@ 3,16 3,16 @@
This repository contains my very opinionated (and possibly overengineered)
robots.txt generation.

`lists` contains lists of user agents to disallow at root
`lists/` contains lists of user agents to disallow at root:
 * .txt files, line by line listings of user agents
 * empty lines are ignored
 * lines starting with # are ignored

`bases` contains site specific base robots.txt files
 * base robots.txt files in the robots.txt format
`bases/` contains site specific base robots.txt files:
 * in the robots.txt format
 * names represent the domains they're served at

`generate.sh` is a bash script for generating output robots.txt files
`generate.sh` is a bash script for generating output robots.txt files:
 * arg $1 is a required path to the out file
 * arg $2 is an optional path to a base file