IMHO, that's not part of learning the language. You don't really even need half of the packages you mentioned unless you don't know the language and, instead, rely upon third-party crutches.
Depends on what your goal is. If your primary purpose for learning python is to be able to use numpy and pandas, then you should probably start learning them from day 1. Learning how to inefficiently implement parts of pandas in pure python on an ad-hoc basis is a waste of time for most people compared to just spending that time learning how to efficiently use pandas.
