"For many companies it matters a lot if you can announce something in time for Christmas."
It will be done by xmas or it won't. I don't recommend "no estimation", but intense estimating won't make you hit xmas. Not estimating may give you several weeks that you could be coding.
Either way you're takin a mighty big risk when you publicly commit to that. Welcome to hell, population: Dev Team. I've known of no major software project that has a public date that hasn't resulted in major delays, feature cuts, or massive overtime to try and hit that date.
"Or you might have to hit a deadline for compliance with a legal contract."
That is also a suitably terrible position to be in. I recommend you don't take such contracts. Again, your estimates will be wrong. Even if you spend months doing them. Then you're in the same spot.
"Or you may be trying to prioritize two different possible products - without a sense of the effort involved you're lacking one of the parameters you need for such an estimation."
Which delivers most business value? I find it farfetched that both are of equal importance.
I have just moved from B2B, where many "deadlines" were largely artificial and driven by sales targets rather than real business need, to online retail, where the deadline is often concrete, immovable, and external (trade show, Olympics, Christmas). In both environments I found it most useful to encourage my team to commit to completion of _the best solution they can manage_ by a business-meaningful date (picked by the team after suitable consideration, of course). In this model the team do not agree to a specific solution or scope, though to pick the target date at the beginning they usually have an initial solution in mind; as they work toward the date, they (working with the customer representatives in the team) have the freedom to cut features or components, or to add new ones, as they grow in their understanding of the feature and the limitations on delivery (legacy code, lack of experience, unclear business needs). I have generally found that in an environment where development teams are trusted (yes I know such environments are far too rare) this produces results that are as good or better than the recipients expected, and almost always on time or nearly so.
Features, budget, schedule. The development team has to have control of one of those three.
Generally schedule is fixed by external factors. Budget is fixed by the existing team size times the schedule. Therefore the corner that makes the most sense for the development team to control is features.
I should have said "sales quotas" rather than targets. Your target to sell £x million in June reflects a real revenue need for the business, but the deadline of 30 June this implies is artificial, and the business would (in most cases) do just as well if you delivered the feature and took the revenue on 1st or 10th July.
Estimating may be a way to figure out what is plausible to do by Christmas. Which may help you scale your desires up or down to have the best deliverable product that you can.
Also in any contract negotiations, someone has to take the risk. The more risk you take, usually the higher margins you can get. (I am currently working on a contract where my willingness to take on all of the risk of not delivering value has improved my likely profit margins 10-fold.) Thus if you're willing to do fixed schedule and/or fixed price contracts, on average you can charge more. But it would be insane to attempt that without being able to estimate your likely costs and schedules.
Sure, that estimation is an intrinsically hard problem. That is why you get to charge extra for having done it. But it is not trivially the wrong business decision to make.
As an extreme example, I offer you SpaceX. Which signs lots of fixed price bids on development tasks that make the average software project look like a cakewalk.
How does this apply in an agency context? I would like to see if this can be applied to my company, but I am a little fuzzy on how this theory can work with paying customers who want to get an idea of cost up front.
"your estimates will be wrong. Even if you spend months doing them. Then you're in the same spot"
If you spend months doing estimates you will never do "nothing else". At the very least, you will have a much learnt a lot about the problem domain, and thus you will have a better overview of the work to do. You likely also will have discovered a few pitfalls to avoid, and you already will have changed the requirements of the project at the moment when it is cheapest: before you have a pile of code.
The FSA (and other regulatory organisations) frequently give a set date when a feature (EG Souped up trade reporting or similar) needs to be in place by.
You claim to be talking about estimation, but I believe that you're actually talking about being given unrealistic, non-negotiable deadlines. (How did we know that the estimate was unrealistic, you ask? We did an estimate.)
It will be done by xmas or it won't. I don't recommend "no estimation", but intense estimating won't make you hit xmas. Not estimating may give you several weeks that you could be coding.
Either way you're takin a mighty big risk when you publicly commit to that. Welcome to hell, population: Dev Team. I've known of no major software project that has a public date that hasn't resulted in major delays, feature cuts, or massive overtime to try and hit that date.
"Or you might have to hit a deadline for compliance with a legal contract."
That is also a suitably terrible position to be in. I recommend you don't take such contracts. Again, your estimates will be wrong. Even if you spend months doing them. Then you're in the same spot.
"Or you may be trying to prioritize two different possible products - without a sense of the effort involved you're lacking one of the parameters you need for such an estimation."
Which delivers most business value? I find it farfetched that both are of equal importance.