It does work out, in fact you could go arbitrarily far in space (and, mandatory, in external time) if you had infinite energy to spend. The acceleration and deceleration phase is almost negligible.
Edit: the faster you go, the slower your own (inertial) time passes. That means the external time passes faster, and the factor grows to infinity the closer you get to C.
In fact, subjectively there is no speed limit. As you go faster, anything around you ages faster, but you yourself won't encounter any speed limit.
But in your inertial reference frame the people on Earth are moving at (near) the speed of light. So they are the ones that should be staying young. Or similarly the planet you are traveling to is actually speeding towards you and you are staying still. This is why the twin paradox is a paradox, because of the reference frames.
You're right that the apparent symmetry is broken by acceleration(s!), and to show that I'd point to Michael Weiss's twin paradox equivalence principle analysis at https://www.desy.de/user/projects/Physics/Relativity/SR/Twin... rather than rewriting it.
There is a subtlety not explicitly raised in the writeup, mainly that in General Relativity metrics do not superpose cleanly, in the sense of getting another solution to the Einstein Field Equations. We do not worry about this in the ultrasimplified twin-paradox model where the spacetime is flat in the sense that the Riemann tensor vanishes everywhere. However, if we want to consider the behaviour of gravitational waves with amplitudes outside of the weak https://en.wikipedia.org/wiki/Linearized_gravity limit, we are in a world of calculational pain.
Physicalizing this subtlety, if our travelling twin is travelling in our neighbourhood of the galaxy, it is probably in for a bumpy ride due to gravitational waves from nearby binary stars https://news.berkeley.edu/2021/02/22/binary-stars-are-all-ar... . We cannot easily extract how bumpy by adding in the uniform pseudogravitational field proposed by Weiss. On the other hand, we probably cannot quantify the effects of gravitational waves at all by simple adapatation of the other strictly Special Relativity analyses at the related Weiss link, https://www.desy.de/user/projects/Physics/Relativity/SR/Twin... (which lists among other the resolution in the minutephysics youtube link you provided above).
You're also right that the problem is one of reference frames. We are not obliged to use that of one twin as the spatial origin. In principle any will do, but some choices have advantages driven by features deliberately excluded from the Special Relativity twin paradox.
Let's consider the "(s!)" tacked on at the end of acceleration. We have not only that of the travelling twin's spacecraft engine, but also that which drives the expansion of the universe.
From within our galaxy we observe a highly spatially homogeneous and isotropic arrangement of extragalactic luminous matter (and cosmic radiation, locally) without distortions in the shapes of distant spiral galaxies that imply a spatially non-flat universe. The metric expansion of this, retaining bulk isotropy, gives us a preferred foliation (Wald's 1984 textbook develops this pp 92-93, but alternatively we could use Weyl's principle). Each twin is free to use a "cosmic fluid" observable (like the dipole-free temperature of the cosmic microwave background, which expands adiabatically), even while accelerating, to determine the https://en.wikipedia.org/wiki/Scale_factor_%28cosmology%29 . For example, each twin could consider the dipole pattern dT/T = v/c where T in the twin's proper time. Each twin can thus determine whether it is the relativistic traveller or not, even if it only wakes up occasionally and only long enough to look at a snapshot of the CMB. The travelling twin thus sees a clear breaking of the Copernican principle along the direction of its travel. Or more precisely, with respect to the bulk flow of matter and radiation in the universe, the non-travelling twin can conclude that it is effectively a Eulerian or comoving observer, while the relativistically-travelling twin cannot.
Moreover, the twins (and any third party) can use "cosmic fluid" observables to determine the scale factor when the twins are together at the start of the travel, and when they (or at least one and the other's remains) are together again at the end.
In this approach there is no paradox at all, there is only the consequences of one twin with a worldline with sections where the proper time is at a higher tilt to the cosmic time than the other twin's. We also avoid the difficulties in attaching a pseudogravitational field to a spacetime where there are gravitational waves of reasonably large amplitude, or relativistic stars and other massive compact objects.
We head into the land of apparent paradox by stripping out evidence of an expanding universe. We must also eliminate evidence of the aging of galaxy clusters through gravitational collapse (including the rate of star formation and the change in abundance of heavy elements). Indeed, we have to arrive in a setting in which neither twin can determine that it has departed from a point at which some reasonable generalization of the Copernican principle applies.
Indeed, the usual formulation of the apparent paradox gets rid of everything but the twins, so that one cannot even use Rindler/Unruh-like observables in flat spacetime, and this really emphasizes the "Special" in Special Relativity.
In that setting, as I said above, relying on the equivalence of being in uniform acceleration (even if it's instantaneous) and being immersed in a uniform (pseudo)gravitational field, is a reasonable way to eliminate the apparent paradox.
There is a related "love triangle" Special Relativity problem where there are three parties: stay-at-home (S), early-outbound-passer (E), and late-inbound-passer (L). None of the parties ever experience any acceleration: they remain eternally in uniform motion, with E & L travelling relativistically.
At our origin, S and E synchronize observe their identical atomic wristwatches coincidentally agree that it is "0". Light-years away, E and L come very close to one another and exchange timestamps showing that coincidentally their identical atomic wristwatches agree. Finally, L and S come very close to one another and compare timestamps from their identical atomic wristwatches. All the wristwatch times are identical to those at the three points in the diagram of the "instant turnaround" version of the twin paradox, we've just turned the travelling twin into two unrelated travellers on different trajectories.
The argument is that this "love triangle" is resolved because E & L are different travellers in uniform motion, so all parties must combine the times acquired in two different reference frames (E's and L's) to compare with the times acquired in S's reference frame. The further argument is that this duplicates the "instant turnaround" version of the twin paradox if we can have the travelling twin change direction without acceleration.
Firstly, we can still solve this with a pseudo-gravitational field popping up at the moment E & L exchange timestamps. It's no more of a coincidence than the identical timestamp when S & E are close.
Secondly, it's not clear that the paradox remains interesting in this case, because there is no expectation that S & L should be the same age when they are close to one another again. They aren't twins. Unless we add in accelerations, there is no way by which S, E, and L could all have been born at close to the same location in spacetime.
Thirdly, it's unclear that there can be an instant turnaround without acceleration. A couple flavours have been explored here and there.
One involves a slingshot around a star to change directions from away to towards the stay-at-home twin. In this picture the travelling twin is always in free-fall. But here we are substituting real gravitation (that of the star) from pseudo-gravitation. We've moved from everywhere-flat Minkowski space -- the spacetime of Special Relativity -- to something closer to Schwarzschild spacetime, which is only asymptotically flat. Moreover, we are using the near region of Schwarzschild to accomplish the slingshot.
Another substitutes the open flat Minkowski space with one in which there is a compact spatial dimension that curls back on it self. A universe with the geometry of a cylinder with infinite height and small circumference, or a torus, or a sphere would do. The cylindrical case has been explored recently : https://doi.org/10.1119/10.0000002 with comparisons to Minkowski space (the spacetime of Special Relativity), §IV (Conclusion) being pithy. Again, I see this as trying to substitute pseudo-geometry with real geometry, an adapted clock-comparison recipe, and a highly privileged frame for the traveller, in order to avoid a non-gravitational acceleration opening the door to a pseudogravitational field arising in the ultrasimplfied and thus strictly Special Relativity problem.
Finally focusing on the latter part of my comment that I'm self-replying to (mostly for my own benefit), we have only done away with one acceleration by the returning twin. We still have the effects from the behaviour of matter in the expanding universe with which to clock S, E and L, removing the remaining paradox if we somehow contrive to have S, E & L expecting to age similarly. If we are abandoning Special Relativity in order to avoid acceleration by the returning without invoking outright magic, why only do it along one spacelike dimension, or by importing a very finely tuned third traveller?
Edit: the faster you go, the slower your own (inertial) time passes. That means the external time passes faster, and the factor grows to infinity the closer you get to C.
In fact, subjectively there is no speed limit. As you go faster, anything around you ages faster, but you yourself won't encounter any speed limit.