Basically yes, and this is reasonable from a security perspective, otherwise every smart contract would basically have a trusted administrator.

But in many cases you do want some ability to upgrade the code, and then you simply program in that ability by creating a repointable proxy with whatever authorization logic you want.