It's not even only that, when you switch to another worktree you catch your language server with its pants down, and some tooling e.g. test watchers are bugging out as well.
I just always found it cleaner to have several directories on the rare occasion I need to work on several branches almost in parallel.
And let's face it, most of us don't work in a repo that is 100GB or above. Even if it was a 10GB I wouldn't care one bit and can easily have 20 copies of it temporarily.
The most recent episode[1] of the Huberman Lab Podcast[0] talked about this briefly, in the context of alertness and focus:
...I haven't figured out yet how to develop a workstation where the computer is above me. I think the only way to really do that is actually to tilt one's body back, but actually that's not a good idea either. They have done studies recording from areas of the brain associated with alertness. Areas like locus coeruleus in the so-called reticular activating system. What they found is that depending on how reclined you are or upright you are, you will decrease with reclining and increase with sitting forward your levels of alertness. So body posture and whether or not your upright or reclining will impact your levels of alertness in the predictable ways. And where you position your eyes, whet her or not your eyes are upright, so to speak, looking up or directly forward or looking down, will dic tate whether or not you are feeling more alert or more sleepy, respectively. So try and arrange a work station or a position of your body in your chair or your standing desk, whatever it is, that allows you to work with a heightened state of alertness.
[0] https://git-scm.com/docs/git-worktree