I guess you're right, but it's more of a curve, though. Once you get to any decent level of complexity, it actually helps to have a framework instead of just going all HTML+CSS. Also it helps having something standard as react (that every web developer should fundamentally understand) than doing your custom stuff if other people will be working on it in the future.
There's a lot to say about the side effects of frameworks but there's a reason why everything converges towards that.
I think it's the other way around, a framework will get you up and running quickly, but then it becomes technical dept, and if your app is complicated you will end up fighting the framework.
If you write something from scratch it will take a while to reach to the abstraction level where you can work fast. But then you have a fully custom abstraction layer that is not a "one size fits all" but custom tailored for your needs.
Good luck with hiring, onboarding, and maintenance of your bespoke solution. Also with your resume when seeking your next gig. For any serious project, ignore community and ecosystem health at your peril. To be clear, we're talking about framework selection, not leftpad vs DIY.
There's a lot to say about the side effects of frameworks but there's a reason why everything converges towards that.