Comments on: Blocking aggressive Chinese crawlers/scrapers/bots

By: michal

michal — Tue, 30 Apr 2024 11:29:18 +0000

Thanks. This solution is faster than plugins like 7G firewall which costs longer to process a htaccess file.

By: Ylva

Ylva — Mon, 19 Feb 2024 04:24:49 +0000

Thank you very much for this solution.
It worked well (client’s website was under attack).

Regards Ylva

By: franck

franck — Thu, 23 Feb 2023 07:27:02 +0000

En 2023 il continue a nous faire chier…

Avez-vous une nouvelle solution, j’ai resté les solutions plus haut mais rien ne fonctionne…

By: Jae

Jae — Thu, 27 Oct 2022 09:46:04 +0000

By: RICHARD

RICHARD — Thu, 01 Jul 2021 11:29:30 +0000

Hi,

Being constantly bombed by the petalsearch bot.

I would be grateful if the following could be checked to ensure that it’s correct.

# BEGIN BOT BLOCK

Options +FollowSymLinks
RewriteEngine On
RewriteBase /
RewriteCond %{HTTP_USER_AGENT} ^.*(Mb2345Browser|MQQBrowser|LieBaoFast|zh-CN|MicroMessenger|zh_CN|Kinza|Bytespider|Baiduspider|Sogou).*$ [NC]
RewriteRule .* – [F,L]

# END BOT BLOCK

Also, looking for a way to create a rewrite rule to send them back to their own originating servers that will raise a red flag to their own servers, hopefully, irritate them to cease crawling website.

Example …

# START REDIRECT BOT BLOCK

RewriteCond %{HTTP_USER_AGENT} (petalsearch) [NC]
RewriteRule ^(.*)$ http://petalsearch.com$1 [R=301,L]

# END REDIRECT BOT BLOCK

By: aeroG

aeroG — Thu, 01 Apr 2021 02:07:21 +0000

Excellent info. I used fail2ban bot filter but the htaccess from the the comments section to block using BrowserMatch has been extremely effective. Thank you and god bless.

By: Mark

Mark — Wed, 03 Feb 2021 17:36:42 +0000

Hi John. This is so helpful, what a great thread! Question please. I have shared hosting with Bluehost.com with a total of five sites, the first site is the primary for the account and the other four are called add on domains. The result is their five sites. My question is, if I add the directives above to my .htaccess to block bots, do I just do the main site? Or do I need to do one for each site? Thank you and the others for great information. Mark

By: topps

topps — Sun, 17 Jan 2021 00:05:32 +0000

you should make the rewrite rule go back to their own site so they can feel the heat they dish out 🙂

By: Paul

Paul — Mon, 04 Jan 2021 14:39:25 +0000

Hi John,

thanks for starting this interesting post & thread. I’m looking for a way to block sogou and baidu from my website (and any other bad Chinese actors), as I don’t offer my services in China and most of my site visits (to one specific, random page) seem to come from sogou. I’m not an expert at this, and I’ve noticed using FTP that I have many layers of htaccess files on my server – above and inside public_html and also one within the folder inside public_html where my site resides. Which one should I add the lines to, and which version should I add? (your 17 March comment seems to override your blog post version, but you’ve also mentioned you’ve amended that because of the comment by Adrian on 20 March)

Many thanks.

By: John Large

John Large — Mon, 19 Oct 2020 14:51:39 +0000

In reply to William Hendricks.

Hi William.

Glad you cracked it. These bots are a pest. Best to block them at the top level of your hosting account to protect any & all websites you host. Be sure to keep an eye on the post. I add new bots to the list as they appear.

The donation towards my caffeine habit is much appreciated 🙂

By: William Hendricks

William Hendricks — Sun, 18 Oct 2020 01:26:07 +0000

Have cup of Joe on me.

Petalbot was being a PITA, tried the deny ip, but after 5 or 6 i gave up. Robots.txt was not only ignored but increased the number of ip’s used.

I pasted the first code into my .htaccess file , which in my case is like an Orangutan doing brain surgery.

But WOW, what a difference, that put the Mojo on Petalbot and a few others

By: Teemu

Teemu — Wed, 01 Apr 2020 07:22:52 +0000

@John Large
Thank you! My site on shared services also got attacked by Chinese bots and I got bandwith limit executed by service provider. With help of these instructions I got situation back on track.