What I know or don't know is irrelevant, because what matters is that your statement rests of bringing in external knowledge/assumptions, so it's clearly not what the commenter is saying (alone).
Indeed, but accusing someone of saying something based on unstated external knowledge/assumptions is the original problem here. They just needed to say words to the effect of "taken with point X what you say implies Y" and it would be fine and much less accusatory.