TL;DR: The existing incentive structure makes it impossible to prevent the creation of large machine learning models like Yudkowsky and others want.
Also, keep in mind that the paperclip maximizer scenario is completely hypothetical.
TL;DR: The existing incentive structure makes it impossible to prevent the creation of large machine learning models like Yudkowsky and others want.
Also, keep in mind that the paperclip maximizer scenario is completely hypothetical.
The AI race is entirely perpetuated by people who wish really hard they were helpless victims of the AI race so they could be excused for continuously perpetuating it in the pursuit of cool demos. Unfortunately, it just isn’t the case. OpenAI to their credit seem to have realized this, hence them not working on GPT-5 yet. - You can see the mask come off on this in Nadella’s “we made them dance” address, where it’s very clear that AI risk simply is not at all a salient danger to them. Everything else is just whining. They should just come out and say “We could stop any time we want, we just don’t want to.” Then at least they’d be honest.
Meta, to their begrudging credit, is pretty close to this. They’re making things worse and worse for safety work, but they’re not doing it out of some “well we had to or bla bla” victim mindset.
You know what happened in scenario 5? We got ten years for alignment. Still pretty impossible, but hey, maybe the horse will sing.
Now, there are plausible takes for going as fast as possible - I am partial to “we should make unaligned AI now because it’ll never be this weak again, and maybe with a defeatable ASI we can get a proper Butlerian Jihad going” - but this ain’t it chief.
I should just clarify, I don’t care about alignment whatsoever. I don’t really care if you disagree; it will only hurt you in the long run.