前段时间，FMFans更换域名以后，访问速度突然变的非常慢。查了nginx日志之后，发现是大量的spider造成的。于是找了个自动根据nginx日志过滤spider的脚本，分享如下:
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| #! /bin/bash | |
| LOGFILE=/var/log/nginx/access.log | |
| PREFIX=/etc/spiders | |
| #日志中大部分蜘蛛都有spider的关键字，但是百度的不能封，所以过滤掉百度 | |
| grep 'spider' $LOGFILE |grep -v 'Baidu' |awk '{print $1}' >$PREFIX/ip1.txt | |
| # 封掉网易的有道 | |
| grep 'YoudaoBot' $LOGFILE | awk '{print $1}' >>$PREFIX/ip1.txt | |
| #封掉雅虎 | |
| grep 'Yahoo!' $LOGFILE | awk '{print $1}' >>$PREFIX/ip1.txt | |
| # 过滤掉信任IP | |
| sort -n $PREFIX/ip1.txt |uniq |sort |grep -v '192.168.0.' |grep -v '127.0.0.1'>$PREFIX/ip2.txt | |
| # 如果一小时内，发包不超过30个就要解封 | |
| /sbin/iptables -nvL |awk '$1 <= 30 {print $8}' >$PREFIX/ip3.txt | |
| for ip in `cat $PREFIX/ip3.txt`; do /sbin/iptables -D INPUT -s $ip -j DROP ; done | |
| /sbin/iptables -Z // 将iptables计数器置为0 | |
| for ip in `cat $PREFIX/ip2.txt`; do /sbin/iptables -I INPUT -s $ip -j DROP ; done |
