OK, how about the theory that mens shirts have buttons on the right side so that the swordsman would unbutton his jacket with his left hand while drawing with his right?
So basically, if the action is easier to do with the right hand, it's self-explanatory, and if it's easier to do with the left hand, it's because you used to do it while using a sword in your right hand. We can explain an infinite number of chiralities now!
I have better dexterity in my right hand, so I find it easier when the button is on the right side. My left hand only needs to hold the hole steady, while the right hand need more flexibility