Google wrote BERT and they provide technical support to the FB Pytorch TPU port so it's not entirely surprising. RoBERTa, (Fb's variant) would be a good candidate to test it with.

We only see code when customers open-source it or otherwise explicitly share it with us. We are directly in touch with several customers who are using the PyTorch / TPU integration, so we hear feedback from them, and we also run a variety of open-source PyTorch models on Cloud TPUs ourselves as we continue to improve the integration.