Rationalist Infighting!

tldr; one of the MIRI aligned rationalist (Rob Bensinger) complained about how EA actually increased AI-risk long-run by promoting OpenAI and then Anthropic. Scott Alexander responded aggressively, basically saying they are entirely wrong and also they are bad at public communications! Various lesswrongers weigh in, seemingly blind to irony and hypocrisy!

Some highlights from the quotes of the original tweets and the lesswronger comments on them:

• Scott Alexander tries blaming Eliezer for hyping up AI and thus contributing to OpenAI in the first place. Just a reminder, Scott is one of the AI 2027 authors, he really doesn’t have room to complain about rationalist creating crit-hype.

• Scott Alexander tries claiming SBF was a unique one off in the rationalist/EA community! (Anthropic’s leadership has been called out on the EA forums and lesswrong for a similar pattern of repeated lying)

• Rob Bensinger is indirectly trying to claim Eliezer/MIRI has been serious forthright honest commentators on AI theory and policy, as opposed to Open-Phil/EA/Anthropic which have been “strategic” with their public communication, to the point of dishonesty.

• habryka is apparently on the verge of crashing out? I can’t tell if they are planning on just quitting twitter or quitting their attempts at leadership within the rationalist community. Quitting twitter is probably a good call no matter what.

• Load of tediously long posts, mired with that long-winded rationalist way of talking, full of rationalist in-group jargon for conversations and conflict resolution

• Disagreement on whether Ilya Sutskever’s $50 billion dollar startup is going to contribute to AI safety or just continue the race to AGI.

• Arguments over who is with the EAs vs. Open Philanthropy vs. MIRI!

• Argument over the definition of gaslighting!

To be clear, I agree with the complaints about EA and Anthropic, I just also think MIRI has its own similar set of problems. So they are both right, all of the rationalists are terrible at pursing their alleged nominal goals of stopping AI Doom.

I did sympathize with one lesswronger’s comment:

More than any other group I’ve been a part of, rationalists love to develop extremely long and complicated social grievances with each other, taking pages and pages of text to articulate. Maybe I’m just too stupid to understand the high level strategic nuances of what’s going on – what are these people even arguing about? The exact flavor of comms presented over the last ten years?

Eliezer is trying to get around that with some weird conditions and game on the prediction market question:

This market resolves N/A on Jan 1st, 2027. All trades on this market will be rolled back on Jan 1st, 2027. However, up until that point, any profit or loss you make on this market will be reflected in your current wealth; which means that purely profit-interested traders can make temporary profits on this market, and use them to fund other permanent bets that may be profitable; via correctly anticipating future shifts in prices among people who do bet their beliefs on this important question, buying low from them and selling high to them.

I don’t think that actually helps. But Eliezer is committed to prediction markets being useful on a nearly ideological level, so he has to try to come up with weird complicated strategies to try to get around their fundamental limits.

A Korean developer named Sigrid Jin—featured in the Wall Street Journal earlier this month for having consumed 25 billion Claude Code tokens—woke up at 4 a.m. to the news. He sat down, ported the core architecture to Python from scratch using an AI orchestration tool called oh-my-codex, and pushed claw-code before sunrise. The repo hit 30,000 GitHub stars faster than any repository in history.

Considering how one of the major use cases of llm coding agents is laundering open source and copy left, this is some well deserved payback to Anthropic imho.

This article is where I was getting the 2018 date from: https://theweek.com/feature/1020838/jk-rowlings-transphobia-controversy-a-complete-timeline

And since in that 2018 incident Rowling was trying to backpedal/downplay it, I assume before that she was keeping the mask firmly on.

Yeah. And it is not like Eliezer usually holds himself back from throwing out hot takes or inserting himself into conversations he is tangentially relevant to, so the silence is conspicuous in this case.

Being the kind of writer I am, whenever this comes up I am tempted to suggest ways it could have been done better.

The premise kind of does work in a setting like Harry Potter, the wizarding world is insular enough that a clever kid could bring in some new ideas. The problem is Eliezer wanted to throw in too many shortcuts. Its not enough for creativity with transmutations to give the protagonist a small edge, transmutation is made into the ultimate all-purpose spell so the protagonist can exploit it easier. The protagonist isn’t just moderately better at Patronus with some muggle psychology, his patronus can kill dementors. And the philosopher’s stone is changed into some ancient atlantean super-magic, because fuck wizards ever inventing anything, and also instead of some moderate rate of its typical mythological powers it is super transmutation.

But, first, I am not glazing the work of Rowling, even indirectly, no way, no how.

Rowling went mask off transphobe in 2018, HPMOR finished in 2015. So I won’t blame Eliezer for not picking a different fandom at the time. Eliezer has actually said moderately supportive comments, including of using people’s preferred pronouns (we’ve mocked another lesswronger for writing long screeds complaining about this). In general, I think the average lesswrong attitude towards trans people is better than the average American’s attitude… but that is because the bar is in hell. But yeah I’ve seen plenty of shitty takes towards trans people on lesswrong.

Second, HPMoR was cult shit all along, not meant to teach science but to sow distrust of scientists under the glossy sheen of being able to name the six quarks.

Yep. And it didn’t even stick to its premise of “try to do science to magic and compare muggle scientifically gained knowledge to magic” and instead went into some Ender’s game pastiche followed by Death Note style “I know you know I know” plotting, then Harry gets handed all the magical power handed to him at the end of the story thanks to Dumbledore following some insane combination of prophecy.

The attitude that you can substitute a bunch of cheap tricks and hacks to get around fundamentally difficult problems reminds me of the techbro attitude that leads to stuff like pushing fundamentally non-viable technologies (like Theranos or the LLM boosters) and of DOGE trying to asking an LLM how to cut the DEI.

Your first point is true… with the key words being “shorn of context”. When you look at how many ratfics go in that direction your second and fourth points become problems.

As to your fourth point… the techbro billionaires like Elon or Peter Thiel do like referencing fiction (often in hamfisted or ignorant ways that makes me think a bit of fandom gatekeeping actually is good sometimes… i.e. naming your surveillance company palantir, or naming one of your kids a nonsensical WH40K reference). So I wouldn’t entirely neglect the possibility of rationalist managing a bit of inspiration to the billionaires in between the bootlicking. And although there may not be a magical “coup the government” power in real life, the influence they are trying to focus on themselves and harness is still worrying.

A lesswronger asks are we rationalfic protagonists the baddies? https://www.lesswrong.com/posts/FuGfR3jL3sw6r8kB4/richard-ngo-s-shortform?commentId=uDuzmfMEvEqpyApLh

tldr; rationalfic has a very common trend of the protagonist gaining and using overwhelming power to radically reform the world. This is almost (with a few notable exceptions) portrayed as clearly unambiguously good thing.

My take: Don’t get me wrong, the Wizarding World (for example), as canonically portrayed needs some very strong reforms if not an entire revolution. But rationalfic almost never portrays the slow hard work of building support networks and alliances and developing a materialist theoretical understanding of how to reform society, as opposed to a lone (or small friend group) rationalist hero finding some overwhelming magical or technological advantage they can use to single-handedly take control and use their rationalist intellect to unilaterally fix everything. Part of it is the normal disconnect of fiction to the real world were it is more narrative satisfying (and easier to write) to have a central protagonist the solves the major problems or is at least directly involved with them, and rationalfic involves that protagonist gaining even more agency than they canonically do. The problem is that rationalist take this attitude back into real life, and so end up idolizing mythologized techbro billionaires or venture capitalist or the myth of the lone genius scientist/inventor.

Also, quality sneer in the replies, “rational” teletubbies: https://tomasbjartur.bearblog.dev/rational-teletubbies/

Is switching to pdm as easy as installing it, then initializing pdm and remaking the pyproject.toml by adding all the dependencies with pdm? (If I’m understanding right, this basically seems like the same workflow as poetry?)

There is poetry for package management. Apparently uv is substantially faster at solving package dependencies although poetry is more feature rich. (I’ve only used poetry, so I know it is adequate, but I have had times I’ve sat there for minutes or even tens of minutes while it worked through installing all the right versions of all the right libraries.)

Yeah. When it comes down to it, the libs think the problem with Trump isn’t the fundamentals of what he is doing, it is that he is doing it without decorum or checking all the legal boxes or saying the usual lib pabulum to justify American imperialism. Skipping the legal checks and decorum is also bad, but in fact kids in cages was horrible when Obama was doing it the “right” way.

I wonder if one of the reasons Pete Hegseth is going so hard after Anthropic is that he and other idiots in the Pentagon unironically believes shit like AI 2027 and so wants to soft nationalize the frontier companies so to control the coming AGI. Considering that one of the uses the DoD allegedly wants LLMs for is fully autonomous weapons that at the very least have a very distorted view of what the technology is capable of. Or they want an accountability sink so they can kill people with even less accountability. …probably both.

I find it darkly hilarious that the doomer crit-hype is finally coming around to bite them, not in the form of heavy handed shut-it-all-down regulation to stop skynet, but in the form of authoritarian wackos wanting to make sure they are the ones “in charge” of skynet.

Did you know that same week this fight was going public Anthropic gave up on their “Responsible Scaling Policy”? (Well, technically they changed to a new version of their RSP that was even more empty and toothless.) To be fair the RSP was basically doomer crit-hype safety theater (“we have a plan for if our AI is so dangerous it is a catastrophic risk”), but if they actually followed it, they would have to stop releasing new models (or else unhype their model’s capabilities), so it was obvious they would abandon the RSP at some point (even many lesswrongers and EAs expected this).

I would bet that the timing of ditching the RSP was a deliberate marketing strategy to mask one ethical backslide behind an ethical stand… except only booster and doomers even remotely expected the RSP to have any meaning in the first place. Still, comparing number of lesswrong, EA, and /r/singularity discussions on RSP v3 compared to discussions on the fight with the DoD, I think they did succeed in minimizing what little criticism they got.

That was their original pitch against openAI

So yeah. People on places like /r/singularity were starting to get skeptical of Anthropic’s claims about ethics, but after this current saga I see loads of comments glazing them and praising them, so mission success.

I wonder if Hegseth realizes he has basically given Anthropic’s marketing team exactly what they want?

I agree this is an important development in this continued saga, but as I said in the main thread, I really don’t like this article’s framing (to the point I wouldn’t be surprised if the author is MAGA or at least prone to sanewashing MAGA).

Reposting what I wrote in the other thread:

Anthropic CEO Dario Amodei picked a major fight with the Department of Defense last month, asserting that his company’s AI models couldn’t be used for mass surveillance of Americans or direct autonomous weapons systems.

As to who picked a fight with who, the DoD wanted to change the terms of their contract, to which Anthropic apparently compromised on every term except for mass surveillance of Americans (fuck the rest of the world I guess) and fully autonomous weapons (cause a human clicking “yes to confirm” makes slop-bot powered drones so much better). This wasn’t good enough for this authoritarian strongman administration, so Pete Hegseth took the fight public with tweets first. So the article framing it as Anthropic “picking a fight” is a bullshit framing. I mean, they did kind of bring it on themselves hyping up their slop machine like it was a sci-fi AGI, but they didn’t start the fight.

For one, “it’s 100 percent in the government’s prerogative to set the parameters of a contract,” Snell & Winter partner Brett Johnson told Wired, effectively meaning there may be very little chance of an appeal.

So they find a quote about contracts, but a Supply Chain Risk isn’t just the DoD deciding on contracts, it is a specific power that has specific mechanisms set by legislation. If (and it is a big if with the current Supreme Court’s composition) the court actually considers the terms set out in the legislation (including, most problematically for the DoD, a risk assessment and consideration of less intrusive alternatives), I think the DoD loses. Of course, the SC has all too often been willing to simply defer to the executive branch’s judgement, even if the process for the judgement was “Trump or one of his underlings made a choice on a spiteful or idiotic whim, announced it on twitter, and the departments underneath them rushed to retroactively invent a saner rationalization”. If the DoD decided to just end the contract (without all the public threats of SCR or invoking the Defense Production Act) Anthropic wouldn’t be in a position to sue and this drama wouldn’t have been as publicized in the first place.

But the lawsuit itself takes a dramatically different tone.

Yeah because one set of a language is a CEO trying to grovel and backtrack on one of the rare few ethical commitments he has ever made (edit well actually Anthropic has made lots of ethical commitments, many of which they’ve already folded on, this is one of the only ones they’ve held against pressure and one of the only ones the media/public might actually expect them to hold to because the fight was so dramatically public), and the other is making a court case about the actual law.

If the DoD accidentally pop the AI bubble by triggering a cascade when Anthropic runs into issues; then later the DoD loses the court case in a humiliating enough way; then DoD loses a civil case with the money going to pay the debts owed in Anthropic’s bankruptcy proceedings, and the American public blames all of (without letting one shift the blame to the other) the Trump administration, the Republican party, the parts of the Democrat that acted as pathetic enablers, and the tech ceos for the following economic depression… I would count that as a relative win?

The specific article’s framing pisses me off…

Anthropic CEO Dario Amodei picked a major fight with the Department of Defense last month, asserting that his company’s AI models couldn’t be used for mass surveillance of Americans or direct autonomous weapons systems.

As to who picked a fight with who, the DoD wanted to change the terms of their contract, to which Anthropic apparently compromised on every term except for mass surveillance of Americans (fuck the rest of the world I guess) and fully autonomous weapons (cause a human clicking “yes to confirm” makes slop-bot powered drones so much better). This wasn’t good enough for this authoritarian strongman administration, so Pete Hegseth took the fight public with tweets first. So the article framing it as Anthropic “picking a fight” is a bullshit framing. I mean, they did kind of bring it on themselves hyping up their slop machine like it was a sci-fi AGI, but they didn’t start the fight.

For one, “it’s 100 percent in the government’s prerogative to set the parameters of a contract,” Snell & Winter partner Brett Johnson told Wired, effectively meaning there may be very little chance of an appeal.

So they find a quote about contracts, but a Supply Chain Risk isn’t just the DoD deciding on contracts, it is a specific power that has specific mechanisms set by legislation. If (and it is a big if with the current Supreme Court’s composition) the court actually considers the terms set out in the legislation (including, most problematically for the DoD, a risk assessment and consideration of less intrusive alternatives), I think the DoD loses. Of course, the SC has all too often been willing to simply defer to the executive branch’s judgement, even if the process for the judgement was “Trump or one of his underlings made a choice on a spiteful or idiotic whim, announced it on twitter, and the departments underneath them rushed to retroactively invent a saner rationalization”. If the DoD decided to just end the contract (without all the public threats of SCR or invoking the Defense Production Act) Anthropic wouldn’t be in a position to sue and this drama wouldn’t have been as publicized in the first place.

But the lawsuit itself takes a dramatically different tone.

Yeah because one set of a language is a CEO trying to grovel and backtrack on one of the rare few ethical commitments he has ever made, and the other is making a court case about the actual law.

It’s so fucking pathetic, he can’t even hold onto the very narrow and weak stand (because he left open a lot of things with Anthropic’s “two red lines”) he took without trying to backpedal and grovel.

your mode of analysis is closer to erotic Harry Potter fan fiction

To give Gary Marcus credit here, HPMOR may not be erotic, but many of Eliezer’s other works are erotic (or at least attempt to be), the most notable being Planecrash/Project Lawful which has entire sections devoted to deliberately bad (as in deliberately not safe, sane, consensual) bdsm.

Eliezer tried to promote/hype up Project Lawful on twitter, maybe hoping it would be the next HPMOR, but it didn’t quite take. Maybe he failed to realize how much of HPMOR’s success was being in the popular genre of Harry Potter fanfic (which at the time had crap like Partially Kissed Hero or Harry Crow as among its most popular works), and not from his own genius writing.

lib brains have a hard time comprehending that there can be multiple bad guys at a time, or that America was in fact a neocolonialist imperialistic empire even before Trump took over and took off the mask.