Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

You can trivially block Cloudflare crawl via robots.txt. You don't need to buy Cloudflare's bot protection -- this is not a malicious bot.

https://x.com/CloudflareDev/status/2031745285517455615

(Disclosure: I work for Cloudflare but not on this product. I get pretty tired of the conspiracy theories TBH.)

 help



Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: