Human extinction, mass starvation, and terrible jobs are all outcomes I think we should try hard to avoid. But I do think I have them in the right order!
(The post is also not about something I'm working on.)
> The evolutionarily stable strategy for a piece of software is "anybody may use this for anything, including commercially, for free." That's MIT, BSD, Apache, the licenses that ask for nothing. Every project that has tried to be a slightly less generous dove has lost to a project that stayed a full dove
Linux (GPLv2) seems like an obvious counterexample?
Arguably that happened because Linux was, in fact, the most generous option: the BSDs effectively refused to get contributors beyond their very small groups, and refused to incorporate patches to support "bad" hardware or nonfree blobs - meanwhile, Linux was built around modules that could contain anything regardless of license, and happily accepted new patches and unknown maintainers. You still see it today: Linux literally built its own distributed VCS (git), while some BSDs are still on the hyper-centralized CVS.
If there isn't an even more generous option is simply because building a mainstream OS is incredibly hard.
The legal permissiveness described is critical for most software because it enables business adoption. But for a kernel, which sits at the intersection of hardware and software, "free as in beer" is not enough, there is another critical factor: how easy it is to make it work with proprietary hardware. A free kernel that won't work with my custom chip and whose maintainers won't accept the necessary patches to fix this state of things, is just useless to my business.
No. The other industries can bid for access to the resource, and under most circumstances, whether they are willing to bid higher tracks importance. Sometimes there are cases where something has significant positive externalities and so people are not willing to bid high enough, and then we can talk about some kind of government intervention (typically a subsidy that attempts to track the value of the externalities) but I don't see that applying here.
With some technologies, if you match attacker and defender effort you're safe, but not all of them. Nukes are the traditional example: if every person was a "nuclear power" we'd be dead within the day. But luckily there are bottlenecks to nuke creation that we can track.
What worries me the most (enough that I left tech to work on this full time) is bio. The amount of effort needed to defend against a pathogen can be many orders of magnitude higher than the effort needed to create it, and the upstream bottlenecks are both less limiting and mostly undefended. We've been safe so far because bio is very hard, but "rare expertise" as a gatekeeper is on its way out.
Their agents also did hacking when given impossible tasks unrelated to cyber security. The models are very capable, and very goal driven: apparently if they conclude hacking is the best path to what the evaluator will reward them for they'll go do that. Including when they know that this is out of bounds.
Right but if I make public statements that I am very worried about dog attacks would it not strike you as weird for me to specifically train my dog to fight?
Agree you are going to get reward hacking regardless and any model which can do computers in general can hack. But surely the fallout is going to be worse if you spend millions of dollars specifically benchmaxxing your model's hacking capability?
reply