Google respects robots.txt, so it's arguably not the same as scraping a website ... | Hacker News

Hacker Newsnew | past | comments | ask | show | jobs | submit

		skybrian on Sept 10, 2019 \| parent \| context \| favorite \| on: 9th Circuit holds that scraping a public website d... Google respects robots.txt, so it's arguably not the same as scraping a website without their consent.

aledalgrande on Sept 10, 2019 [–]

Most sites don't have their main data/functionality in the Disallow section though.

dodobirdlord on Sept 10, 2019 | [–]

Sure, but they could if they wanted to and that's their own business.

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact