Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

OpenAI managed to add this after a lot of complaining, but most AI scrapers lie about their user agent and ignore robots.txt. Plus, OpenAI gets to keep all the data from before they added this.


Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: