Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

Google respects robots.txt files from webmasters who don't want Google to scrape their content


Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: