Basically I just tell them the truth - that the project doesn't seem to be a good fit for my business right now. If they insist on knowing why, you can tell them the risk vs profit factor is too high for you at this stage. If they want to know what that means, you can just say that you don't think you can give a good estimate of the total project time/cost so you don't feel comfortable bidding on it.
If they're reasonable, they may agree to just go hourly then upon hearing a reasonable explanation. Sometimes they'll try to negotiate on the price. I tend to be very wary of clients that try to negotiate rates though because even if you get your rate you'll find yourself later arguing about hours, features, etc. Your rate can effectively be lowered if they pressure you into feeling that you are not working fast enough and bill 1 hour for 2 hours of actual work. People who like to haggle tend to haggle over every detail to get a bargain - not just price.
Wish I had seen this post just an hour ago :-) I turned down a prospective client for that very reason. I said basically the same thing, but you phrased it better than I did.
If they're reasonable, they may agree to just go hourly then upon hearing a reasonable explanation. Sometimes they'll try to negotiate on the price. I tend to be very wary of clients that try to negotiate rates though because even if you get your rate you'll find yourself later arguing about hours, features, etc. Your rate can effectively be lowered if they pressure you into feeling that you are not working fast enough and bill 1 hour for 2 hours of actual work. People who like to haggle tend to haggle over every detail to get a bargain - not just price.