I'll second the points about yale culture. Honestly, if you're worried about being A downside would be continuing to be the “too-intense guy” I come off as at my high school, and potentially continuing to feel slowed down by my peers." then if you're as technically / mathematically oriented as you claim, outside of certain parts of mit, caltech and other places with a large engineering culture, you'll continue to be viewed as such.
A simple, but in retrospect insightful bit of advice that I ignored (but shouldn't have) when choosing a university is the following: go through the course catalogue and figure out what percentage of all the classes you would certainly find interesting, all other considerations being equal, choose the school with the best percentage!
I'm into less technical subjects almost as much as science as engineering; most of the "too-intense" stuff I think comes from me being overall achievement and knowledge focused. So, I'd hopefully be far less out of place at Yale than at my high school, or even Tufts.
Thanks for the advice on the course catalog - I'll definitely do that.
A simple, but in retrospect insightful bit of advice that I ignored (but shouldn't have) when choosing a university is the following: go through the course catalogue and figure out what percentage of all the classes you would certainly find interesting, all other considerations being equal, choose the school with the best percentage!