There are a few ways around this, depending on the use.
The owner of a wallet only cares about transactions relevant to the wallet. From the beginning, Bitcoin has had a method called Simplified Payment Verification (SPV) that uses block headers and only those transactions needed for the wallet to work: