Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

Yes, but the difference is that the term "scraping" also targets things like automatically generating RSS feeds from HTML pages, which is not covered by robots.txt.


I thought robots.txt covered all automated, programmatic access by third parties where a bot slurps stuff and follows links, without splitting hairs about it.

But what do I know, the young whippersnappers will just word lawyer me to death, so I better shut up and go away.




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: