All containers use user namespaces, but only rootless containers require unprivileged user namespaces. Unprivileged user namespaces didn't have a great security record for the first few years, but vulnerabilities are relatively rare these days.

Running podman with SELinux enforcing (the default) and with "--security-opt=no-new-privileges" combined with running applications as non-root inside their containers should further reduce the security risk. You could also disable unprivileged user namespaces inside the containers if you want, which would mean that exploiting unprivileged user namespaces would first require arbitrary code execution on the host.