Yes it reduces the gain you get from accelerating in a deep gravity well, so you lose efficiency in your transfer, plus you need to actually get there first. It's a lose-lose situation.

If there is anything to pick up from the moon it makes more sense to deliver it to LEO first and continue from there.