> A pastime I have with papers like this is to look for the part in the paper where they say which models they tested.
My pastime (not really) in HN submissions like this is to look for the comment where someone complains about the models used because they aren’t the literal same model and version the commenter has started using the day before.
It’s always “you can’t test with those models, those are crap, the ones we have now are much better”, in perpetuity. It’s Schrödinger’s LLM: simultaneously god-like and a piece of garbage depending on the needs of the discussion. It’s beyond moving the goalposts, it’s moving the entire football field. It’s a clear bad faith attempt to try to discredit any study the commenter doesn’t like. Which you can always do because you can’t test literally everything.
The GP's criticism as I read it is about paper authors not making it particularly easy to reproduce their findings.
For a long time I have criticized this too, especially for software projects, or papers that deal with machine learning models. If the things described in a paper are not reproducible, then it's basically worthless. Similar to "it works on my machine" in software engineering. Many paper authors are not software engineers, and often neither are they experts in the tooling they should be using to make their research reproducible. If this is a problem for a research team, then please, hire an engineer to ensure reproducible. It doesn't help anyone to remain ignorant towards the reproducibility issue and only shows lack of scientific discipline. Reproducibility should be on the mind of any serious researcher and there should be lectures about how to do it at universities.
Firing off glib criticism that amounts to “No study on AI is valid beyond the release cycle of the models tested,” feels like the unconscious self-protection reflex we all default to when facing cognitive dissonance. It seems like it’s only easy to spot when someone you disagree with is doing it.
To me, it almost feels like a partisan political thing.
My pastime (not really) in HN submissions like this is to look for the comment where someone complains about the models used because they aren’t the literal same model and version the commenter has started using the day before.
It’s always “you can’t test with those models, those are crap, the ones we have now are much better”, in perpetuity. It’s Schrödinger’s LLM: simultaneously god-like and a piece of garbage depending on the needs of the discussion. It’s beyond moving the goalposts, it’s moving the entire football field. It’s a clear bad faith attempt to try to discredit any study the commenter doesn’t like. Which you can always do because you can’t test literally everything.