Timers aren't necessarily even a requirement to exploit Spectre: https://news.ycombinator.com/item?id=22831067. It's pretty hard to protect against this in general unless you generate retpolines.