It's fast enough for now, so investing more time in making it faster would just be spinning wheels.
I haven't done detailed profiling, but the geoip lookup is pretty fast. I think more time is spent picking IPs to return (weighted from a list of sometimes thousands of IPs) and likely more time than that is in the underlying DNS library (which hopefully will get better over time without me doing anything!).
Both the Perl and the Go versions are optimized more for developer time, correctness and robustness over raw performance. I only have so much time to work on it and lots of you depend on it working, so I think those are appropriate trade-offs.
If it was a full-time job more than a hobby maybe it'd make sense to do the work in C instead of Go (but probably not).
Even if you do geo lookup on a per /24 basis, surely that doesn't detract that badly ?