But it sounds like in this case the "morality" that capitalism doesn't account for is basically just someone saying "you should be forced to pay me to do something that you could otherwise get for 10x cheaper." It's basically cartel economics.
In isolation that makes sense, but consider that these AIs have been trained on a vast corpus of human creative output without compensating the human creators, and are now being used to undercut those same humans. As such there is some room for moral outrage that did not exist in prior technical revolutions.
Personally, I think training is "fair use", both legally and practically -- in my mind, training LLMs is analogous to what happens when humans learn from examples -- but I can see how those whose livelihood is being threatened can feel doubly wronged.
The other reason I'm juxtaposing Capitalism and morality is the disruption AI will likely cause to society. The scale at which this will displace jobs (basically, almost all knowledge work) and replace them with much higher-skilled jobs (basically, you need to be at the forefront of your field) could be rather drastic. Capitalism, which has already led to such extreme wealth inequality, is unsuited to solve for this, and as others have surmised, we probably need to explore new ways of operating society.
