If you've reduced the number of syscalls, the mitigations almost don't matter. Turning off the mitigations is more important if you're using the kernel stack for something like a tuned HAProxy installation which is mostly syscalls.