I am not sure we are disagreeing on much here. Yes, non-covexity is major problem for optimization. And yes, very simple problems could be analyzed in high dimensions. But many problems are not simple, including understanding general structure of the information/data flow through the network as well as non-convex optimization itself. The work on infinite width networks is very interesting and is getting novel insights. Many theoretical CS researchers are working on understanding of deep neural networks. I should have rephrased the sentence from "Somebody needs to do this" to "It would be nice if we could make a major progress in this direction".