The problem then is people entering the duress code by accident (if it's similar to the pin) or forgetting it (if it's too different).

Also a criminal might see somebody fumbling with the keypad slower than usual and assume they are entering a duress code.

I generally enter my PIN very quickly , because the movements of my fingers on the keypad is stored in muscle memory.