I ran into similar issues several years ago in setting up "sftp-only" ssh logins to a server. As I recall, the most robust solution involved disabling port forwarding and running the "sftp" ssh daemons in a nearly empty chroot environment.
At the time, I recall finding lots of misleading information on the Internet about restricting ssh, mostly task-oriented "howtos" that overlook the basic principles of operation in order to "get started quickly!"
In my experience, beyond "whitelisting" ssh access in the first place with AllowGroup and related, one should rely more on "deep" kernel-level restriction mechanisms (chroot, BSD jails, sandboxing, etc.) than "shallow" authorization controls ostensibly provided by ssh and/or the login subsystem.
At the time, I recall finding lots of misleading information on the Internet about restricting ssh, mostly task-oriented "howtos" that overlook the basic principles of operation in order to "get started quickly!"
In my experience, beyond "whitelisting" ssh access in the first place with AllowGroup and related, one should rely more on "deep" kernel-level restriction mechanisms (chroot, BSD jails, sandboxing, etc.) than "shallow" authorization controls ostensibly provided by ssh and/or the login subsystem.