This is already a feature in an app called MacroFactor. But there is definitely room for improvement in the field.

One thing that I miss in MacroFactor is that it should have some memory of my previous choice.

Example: If I take a picture of a glass of milk, it always assumes it to be whole milk (3.5% fat). Then I change it to a low fat milk (0.5% fat). But no matter how many times I do that, it keeps assuming that the milk in the photo is whole milk.

Hmm, good idea. I'm adding the ability to add "context" so taking a hpoto shouldn't return the same result.