> LLMs are so intelligent they don't know that a "how many" question is answered with a number.

I think this is to prevent the LLM from giving more details. The evaluation engine can presumably only check short exact answers.