> > We have no idea what crates.io thinks it makes sense to require javascript to look up packages but here we are.
>I've had a similar experience with crates.io:
They do have an API (ps: I built crates.live on top of it). I think they have a very good reasons to block the crawling of their main website. Otherwise, people might abuse it. Actually, they recommend you identify yourself when crawling their API to not limit you. I didn't do it, and found no problem constantly calling their APIs.
So first, "crawling" a website is not abusing it. It's simply using the website and there's nothing wrong with that. Then, i believe that "not found" JSON message was not intended as an anti-scraping measure, but was in their view a meaningful error in the sense that i did not request info about a specific crate so the API responded "not found".
What's weird is no specific Accept header, http://crates.io returns 301 with some HTML, and https://crates.io/ returns 404 with some JSON, while in a browser you get a proper 200 with HTML. I just found that pattern very confusing, but hey maybe i'm just an old dinosaur and that is the future of web development.
>I've had a similar experience with crates.io:
They do have an API (ps: I built crates.live on top of it). I think they have a very good reasons to block the crawling of their main website. Otherwise, people might abuse it. Actually, they recommend you identify yourself when crawling their API to not limit you. I didn't do it, and found no problem constantly calling their APIs.