SEPA Direct Debit is mostly used for recurring payments, and the account owner must authorize the company to use it. Just knowing someone's account number is not enough, you still need that initial authorization.

In fact, in some EU countries businesses are required to publish their bank account numbers in some central, government-run registry. It wouldn't be safe to do, if it could result in having your money stolen.