There's a subtlety here: the "actual" memory unsafety may manifest elsewhere, if the `unsafe` keyword just allows violating constraints that other (possibly-safe) functions rely on. That is, this touches on the whole "boundary of `unsafe` is the module", where functions marked `unsafe` might do perfectly safe things internally but break invariants that other pieces of code assume are true. From the view of "an API" == "an individual function", it is true that incorrect use of `unsafe` code may not risk any memory unsafety in isolation (e.g. one can pass any integer to Vec::set_len and nothing bad will happen in that call), but it is not so true in the more conventional broader view of an API.
Good argument. If you advise someone to use unsafe the responsible thing to do is explain precisely what the consequences are besides "it's faster."