Yep, validation is key. The smartest thing I've heard on this, which has reoriented how I think about this is that the objective function of a piece of software is now more important to get right than the implementation.
> the objective function of a piece of software is now more important to get right than the implementation
That has always been the case. That is why weeks or even months of programming and other project busy work could replace a couple of days of time getting properly fleshed out requirements down.
Agreed, it has always been the case. But I've never thought of it that way so explicitly. And I might argue that the important distinction is that the objective function is programmatically verifiable (which the word "requirements" has not always implied).
No, what is rewarded is "the code has been shown to conform to the given objective function" and especially "that objective function is a good representation of what we are trying to accomplish with this code".
I say this usually about self-driving cars, but the phrase fits here too.
"It doesn't need to be perfect. It just needs to be better than the average human, and humans suck at driving."
- I've taken a controversial new pill that accelerates my brain.
-- So you're smart now?
- I'm stupid faster!
That being said, being stupid faster can work if validation is cheap (and exists in the first place).
Turns out "eh close enough" for AGI is just stupidity in an "until done" loop. (Technically referred to as Ralphing.)