On iOS, at least, it will ask if you want to allow it access to your contacts, and you can just say no.