Why the internet needs crawl neutrality | Hacker News

Hacker Newsnew | past | comments | ask | show | jobs | submit

		Why the internet needs crawl neutrality (fastcompany.com)
		6 points by jedwhite on June 10, 2022 \| hide \| past \| favorite \| 3 comments

groffee on June 10, 2022 [–]

Cloudflare is a large part of this, they actively block all crawlers except for Google.

ColinHayhurst on June 12, 2022 | | [–]

Not true actually [0]. In my experience, at least, Cloudflare has been way more responsive than other big names that I won't call out here.

[0] https://radar.cloudflare.com/verified-bots

SheinhardtWigCo on June 10, 2022 | | [–]

That seems like a strategic vulnerability for Google. Things will get interesting if CF goes into search.

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact