DEVELOPMENT ENVIRONMENT

~liljamo/robots.txt

c8d6ca21955c2194fa482dc509765ad88ec7ff7f — Jonni Liljamo 27 days ago 35b09eb
feat: generate-nginx.sh
2 files changed, 26 insertions(+), 0 deletions(-)

M README.md
A generate-nginx.sh
M README.md => README.md +4 -0
@@ 16,3 16,7 @@ robots.txt generation.
 * arg $1 is a required path to the out file
 * arg $2 is an optional path to a base file

`generate-nginx.sh` is a bash script for generating an nginx if block to block
user agents at that level:
 * prints out an if block with every user agent in the lists


A generate-nginx.sh => generate-nginx.sh +22 -0
@@ 0,0 1,22 @@
#!/usr/bin/env bash

out="if (\$http_user_agent ~* \""

# loop lists
for filename in ./lists/*.txt; do
    # line by line
    while read -r line; do
        # ignore empty
        [ -z "$line" ] && continue
        # ignore comments
        [[ $line =~ ^#.* ]] && continue

        # add to out
        out="$out$line|"
    done < $filename
done

out=${out::-1}
out="$out\") {\n    return 403;\n}\n"

printf "$out"