I personally think the line should be mostly output based. You should be able to train on any copyrighted work by having a single reader license (e.x. purchasing a book or e-book) for that work and no other special licenses. You shouldn't be able to download pirated works for training but you shouldn't need special licenses to train instead of read.

But if your model produces outputs that too closely match their inputs and a company can show it that is a copyright violation and you can be sued for it.