The only real way round this is not to do the emulation on a commodity multi-user system (with all its tremendous advantages!), but in some kind of hardware or bare metal software. Where you can access the original gamepads with the original sub-millisecond latency.
Still can't quite work around the latency imposed by the display, unless you go back to CRT or build your own LVDS driver board.
I had the opportunity to play a Tempest arcade machine recently, and the combination of low latency and the weighted spinning controller was really tangible.
Yes, the best thing that could happen to emulation latency reduction would be the creation of a skeleton OS framework. Something that piggybacks and can use existing Linux or BSD hardware drivers, but puts the emulator in kernel space and lets it manage the hardware as directly as is practical.
I talk about this briefly in my infeasible section. But I'm mostly referring to myself there: I just don't have the skill and the time to write my own real-time OS for this.
However, I do hope that one day we see a serious project that isn't just "lightweight Linux" (eg Lakka, etc) or something running on top of DOS.
I presume that 98% of emulator users aren't going to be willing to boot into an "emulator OS" just to play games; so it's going to be a lot of work for a little gain.
In that vein, the ultimate latency reduction would be an entire FPGA emulator, like kevtris' FPGA NES (http://kevtris.org/Projects/console/video/page1.html), but now you're approaching extreme effort for a userbase of maybe ten people :/
Massalin's Synthesis was a research OS designed for low-latency media processing. It's been decades since I read the thesis, so don't trust me to summarize it, but I think it was something like: set up layers in kernel space, but collapse them with runtime code generation (fusing loops), keep switching overhead low instead of buffering a lot, and schedule with a phase-locked loop in software.
That's pretty amazing OS, binary fusion, etc. Something like that would be optimal for current generation hardware, modern CPUs would benefit a lot more from that.
I think the C64 Direct-to-TV by Jeri Ellsworth was quite a success in terms of userbase. Of course the problem with making and selling hardware is increased IP scrutiny.
I think the Raspberry Pi or Beaglebone would make good candidates for baremetal emulation. The Pi has a formidable but barely documented vector processor, and the Beaglebone has the (also under-supported) PRU realtime subsystems which would be ideal for controllers and audio.
Both also have enough ecosystem to make it worth doing. Retropie is already popular, although it's a straight port of the normal userland emulators and doesn't attempt to do anything fancy with the hardware. I think there is a "market" (not the kind that pays money, but a decent userbase) which will happily dedicate a cheap SBC to booting straight into emulators.
Well, debatable on the C64 instance. Indeed, if you can manage to commercially market your design, then you could get a userbase in the thousands. But compare that to the millions of downloads something like Dolphin gets.
And the ARM-based mini systems could work for retro emulators in general, sure. Presuming someone actually writes a true bare-metal OS for them instead of just a lightweight Linux instance. But they're not fast enough for people like me that are obsessive about exacting emulation accuracy. I'm still hoping we'll get to a point where ARM-based computers are taken more seriously and we start getting a real market for products like the Jetson TX1 to compete against budget AMD systems.
All of this stuff involves sacrifices. The more you chase latency, accuracy, etc ... the more you alienate your potential market. Even when we're not looking for money, more users is such a potent motivator to validate that our efforts aren't being wasted.
Last time I used retropie it wouldn't correctly render vibrato effects in the music, so all of FFVI's wonderful score came out flat and awful. That was over a year agohowever, maybe its been updated.
The Raspberry Pi just isn't powerful enough for cycle-accuracy with the SNES. Maybe if a wizard like blargg attempted it by writing the whole thing in pure ARM assembly, but for mere mortals like myself, as awesome as it'd be, it's just not going to happen :(
I wonder if you can't do a jolly good job by having emulators run under Linux with high RT priority and bypassing X by using the framebuffer? You can still have your other cores doing regular multitasking stuff in the background.
It would help even more, certainly. With really good setups, we're battling probably around ~100ms of latency when you combine everything I talked about together. If you were to do as you said, we could maybe drop another 10-20ms of that off.
OLED displays also have very low latency as long as there is no scaler in the way. The Oculus Rift CV1 uses them and has a motion-to-photons latency of 9 ms[0], which includes not just the display but the entire stack.
Yeah, scalers and OSDs are going to be the hardest things to give up.
I will admit ... it's painful using my ZR30w with no scaler. I can't connect my PSP's component out to it, I can't hook up my Wii U, I can't connect my XRGB Mini, my Raspberry Pi, etc ... because none of them output at 2560x1600.
Even worse, the DisplayPort connector stopped working on it (hooray, $1300 monitor quality!!), so all I have is one DVI port left, which can't even output at the 30-bit color depth which was a big part of why I wanted this monitor :(
The ironic thing about OSDs causing input lag is that the very systems we emulate prove that video overlays are trivial to implement without buffering.
Still can't quite work around the latency imposed by the display, unless you go back to CRT or build your own LVDS driver board.
I had the opportunity to play a Tempest arcade machine recently, and the combination of low latency and the weighted spinning controller was really tangible.