아파치 - 나쁜 봇과 크롤러를 차단

아파치 - 나쁜 봇과 크롤러를 차단

 

사용자 에이전트 목록을 보려면 아파치 로그 파일을 검색

cat /var/log/apache2/access.log |  cut -d'"' -f6 | grep -v -E -i "mozilla|chrome|safari|opera" | sort | uniq -c | sort

 

웹사이트에 액세스한 의심스러운 사용자 에이전트 목록이 표시됨

     6 GumGum-Bot/1.0 (http://gumgum.com; support@gumgum.com)
     6 PostmanRuntime/7.19.0
     6 ag_dm_spider v1.0
     7 Microsoft Office Word 2014
     32 Slackbot 1.0 (+https://api.slack.com/robots)
     33 Slackbot-LinkExpanding 1.0 (+https://api.slack.com/robots)
     38 WebexTeams
     41 www.ru
     46 facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php)
     57 admantx-sap/2.4 (+http://www.admantx.com/service-fetcher.html)
     72 Scrapy/2.4.1 (+https://scrapy.org)
     72 ias-sg/3.1 (+https://www.admantx.com/service-fetcher.html)
     76 WhatsApp/2.21.7.14 A
     81 Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)
     83 Clickagy Intelligence Bot v2
     85 ias-jp/3.1 (+https://www.admantx.com/service-fetcher.html)
     92 W3 Total Cache
    105 AccompanyBot
    118 rest-client/2.1.0 (linux-gnu x86_64) ruby/2.7.1p83
    141 -
    159 chimebot
    183 Mediapartners-Google
    374 ias-or/3.1 (+https://www.admantx.com/service-fetcher.html)
    686 ias-ir/3.1 (+https://www.admantx.com/service-fetcher.html)
   1930 CriteoBot/0.1 (+https://www.criteo.com/criteo-crawler/)
   2022 ias-va/3.1 (+https://www.admantx.com/service-fetcher.html)

 

 

이것으로 차단할 사용자 에이전트 목록을 만든다

GumGum-Bot
PostmanRuntime
ag_dm_spider
Scrapy
chimebot

 

 

가상호스트 설정에 추가


<VirtualHost *:80>
        ServerAdmin webmaster@localhost
        DocumentRoot /var/www/html
        ErrorLog ${APACHE_LOG_DIR}/error.log
        CustomLog ${APACHE_LOG_DIR}/access.log combined
RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} (gumgum-bot|postmanruntime|ag_dm_spider|scrapy|chimebot) [NC]
RewriteRule .* - [F,L]
</VirtualHost>

 

참고.

GitHub 프로젝트에서 나쁜 봇 및 크롤러 목록을 제공

https://raw.githubusercontent.com/mitchellkrogza/apache-ultimate-bad-bot-blocker/master/Apache_2.4/custom.d/globalblacklist.conf

https://raw.githubusercontent.com/mitchellkrogza/apache-ultimate-bad-bot-blocker/master/Apache_2.4/custom.d/globalblacklist.conf