It does but the most common JITed language in existence is JavaScript, where the uncertainty is especially pronounced. For example, Python isn't [traditionally] a JITed language but most Python developers know that their code will run on CPython, which is pretty stable and you have a rough idea of which versions it'll run on, whereas something running in a browser can see huge shifts in what is an expensive or cheap operation from user to user over the course of a single day, much less release to release.

Remember when WebKit started JITing regular expressions? That changed the benchmark situation so much that some people called it cheating and not a single line of user code changed. That's not impossible in other environments – I'm sure there's been a case where a shared library update made a big difference – but it's far less common in my experience.