I'd agree that non-shallow reasoning will probably be the biggest challenge to overcome.
> you have no choice but to tell it EXACTLY what you want. Free form language requests are never going to get you there
I don't see any fundamental limitation in this regard compared to the level of detail you'd need to give a human programmer. With existing integrations you can already ask along the lines of "fit a model to this dataset [attachment]" and have it infer a lot of the details, then iterate on it with clarification as you could a human programmer if needed.
> you have no choice but to tell it EXACTLY what you want. Free form language requests are never going to get you there
I don't see any fundamental limitation in this regard compared to the level of detail you'd need to give a human programmer. With existing integrations you can already ask along the lines of "fit a model to this dataset [attachment]" and have it infer a lot of the details, then iterate on it with clarification as you could a human programmer if needed.