So first, "crawling" a website is not abusing it. It's simply using the website and there's nothing wrong with that. Then, i believe that "not found" JSON message was not intended as an anti-scraping measure, but was in their view a meaningful error in the sense that i did not request info about a specific crate so the API responded "not found".
What's weird is no specific Accept header, http://crates.io returns 301 with some HTML, and https://crates.io/ returns 404 with some JSON, while in a browser you get a proper 200 with HTML. I just found that pattern very confusing, but hey maybe i'm just an old dinosaur and that is the future of web development.
So first, "crawling" a website is not abusing it. It's simply using the website and there's nothing wrong with that. Then, i believe that "not found" JSON message was not intended as an anti-scraping measure, but was in their view a meaningful error in the sense that i did not request info about a specific crate so the API responded "not found".
What's weird is no specific Accept header, http://crates.io returns 301 with some HTML, and https://crates.io/ returns 404 with some JSON, while in a browser you get a proper 200 with HTML. I just found that pattern very confusing, but hey maybe i'm just an old dinosaur and that is the future of web development.