Took me a while to figure out who the "we" is (OpenDNS). Might be worth stating it in your comment.

Any idea on the limit for "large" machine based usage?

For examples, I use it with a PHP script that hits it a few hundred times and have not yet had a problem. Should I be expecting one?

We had a 500 cluster web crawler using us. Probably doing about a billion queries a day. They had the option to pay us or switch to Google Public DNS. Guess which they did? :-)

A few hundred, even if a few hundred per minute, is not a problem.

I have operated a rather large crawl cluster on EC2 (several hundred nodes), and I don't see why anyone would point one at a third party DNS service.

we operated for about 6 months using the default resolvers than come with the AMIs, then Amazon contacted us, telling us they wanted us to stop using their recursive nameservers... apt-get install bind, point at 127.0.0.1, redeploy AMI, done (in under 5 minutes).

it isn't as if a crawler cares about an extra 250ms from non-cached entries, and the bandwidth from DNS is trivial* compared to that of downloading pages.

... so why would anyone ever offer to pay you/anyone else for a recursive DNS service? it's a trivial problem...

(* say 32 bytes for the query, 64 bytes for the response... 1B lookups is ~64gb, or $3.50 with Amazon's very expensive bandwidth costs)

250ms matters. That's 1/4 of a second. It matters for a crawler tremendously.

DNS query sizes are wrong. I'd double each. But still, inexpensive from a bandwidth standpoint. I get it.

why would it matter at all, given a crawler hits many pages on the same hostname? the fact the initial request takes 250ms more is meaningless.

when you also take into effect that crawlers are either: bound on sleep() if they're friendly, bound on cpu if doing processing and you have a lot of money for bandwidth, or if you don't have much money, bound on bandwidth.

and given that crawlers tend to be massively parallelised, the DNS query could take minutes and you really still wouldn't care...

(go and read up on Amdahl's law)

Inbound bandwidth to Amazon has been free for a while. Anywhere but Asia and South America, your posited 32 byte query adds up to about $0.36 per billion at their most expensive $0.12/GB tier.

> They had the option to pay us or switch to Google Public DNS.

Do you mean to switch away from Google Public DNS?

> Guess which they did?

How much did it cost them?

No, he's talking about a different DNS - OpenDNS. It's a confusing thread - the top comment is by a guy running OpenDNS. OpenDNS is a DNS service which resolves typos, blocks typo fishers, and sends empty pages to an OpenDNS ad page. They also have a bunch of options and controls, letting you block some sites (useful for employers, schools, and parents).

It sounds kind of sleazy (redirecting "no record" to an ad seems a little off to me), but the guys running it are apparently not. The prejudice against redirects to ad pages is more a result of ISPs who take your money and still give you ads, unlike OpenDNS which is free.

> It sounds kind of sleazy (redirecting "no record" to an ad seems a little off to me)

It doesn't just seem off. It is off: it's a major violation and a large part of why I don't use OpenDNS.

When a name doesn't resolve, I want to see my browser's page for a DNS failure. I don't want to see ads.

It's a free service. Unless your company forces it to use it, OpenDNS is opt-in.

As I said, it makes people mad because ISPs effectively force you to use their DNS, and some use it to serve ads.

Nobody complains about Google serving ads on their pages, but if your ISP inserted ads through some kind of MITM, it would make people pretty angry.

Further, on OpenDNS, you can disable these ads if you so choose.

No, he meant they had the option to start paying OpenDNS or to switch from it to Google Public DNS.

This is all on PowerDNS?

Nope... but we like Bert (the author).