Cloudflare: The Great AI Bot Defender

There are good bots and bad bots! Google good! ChatGTP bad!

Cloudflare is one of my favourite technology companies. Their WAF (Web Application Firewall) is excellent at filtering out malicious traffic, such as DDoS, scanning bots and hacking attempts. The scanning bots are especially bad as they not only grab information from the website, but generally bring down its performance with continual access. With the rise of AI, these bots are not there to archive data for search engines; they want to gather data for machine learning models. A quick look at the logs reveals that a significant percentage of the traffic often comes from automated scripts and bot scanning. For my site, over the past week, I received 367,000 accesses (and over 19 million accesses per year):

Out of this, we can see that around 43.57K (~12%) and 53.63K (~15%) are automated bots. In the following, we see the TikTok crawler bot scanning the site (+TikTokSpider;+ttspider-feedback@tiktok.com):

2025-06-29 00:17:07 10.0.0.106 GET /csharp/nsec_xchacha - 443 - 172.69.176.62…