This and not just for the performance reasons. A Pydantic model represents an I/O boundary of a program. This conveys a lot of information to a programmer reading the code. It would be quite misleading to find it's actually only passed around internally and never used for I/O. A bit like using an int type to store a Boolean value.
On the other hand if I see a dataclass I can tell what it's purpose is by whether it's frozen or not etc.
My rule of thumb is to use Pydantic if I need serialisation, otherwise I default to dataclasses.