Wrong link. this points to the NeurIPS post for this week.

My dad was a bit freaked out by a video version (We’re not ready for super-intelligence)of the “AI 2027” paper, particularly finding two end scenarios a bit spooky: colossus-style cooperating AIs taking over the world, and the oligarch concentration of power one, which i think definitely echoed sci-fi he watched/read as a teen.

In case anyone else finds it useful these are the “Comments as I watch it”, that I compiled for him

Before watching Video Notes:

• AI Only channel with only 3 videos

• Produced By “80000hours”, which is an EA branch (trying to peddle to you the best way to organize 40years * 50 weeks * 40 hours [I love that they assume only 2 weeks of holidays]); which is definitely cult adjacent: https://80000hours.org/about/#what-do-we-do. Mostly appears to be attempting to steer young people to what they believe are “High impact” jobs.

Video Notes:

• The backing paper is a bit of a joke, one “AI 2027”, for reference one of the main authors is very much a “cult member”, Scott Alexander Siskind, author of “Slate Star Codex” and “Astral Codex Ten”.

• Other authors include [AI Futures Project] :

  • Daniel Kokotajlo (podcast co-host of siskind, ex open-ai employee, LessWrong/EA regular)
  • Thomas Larsen (ex MIRI [Machine Intelligence Research Institute = really really culty], LessWrong/EA regular)
  • Eli Lifland (LessWrong/EA regular)
  • Romeo Dean (Astra Fellowship recipient = money for AI Safety research, definitely EA sphere)

• A lot of fluff trying to hype up the credentials of the authors.

• AGI does not have a bounded definition.

• They are playing up the China angle to try and drum up jingoistic support.

• Exaggerating Chat GPT-3 success, by merely citing “users”, without mentioning actual revenue, or actual quality.

• Quote:

  How do these things interact, well we don’t know but thinking through in detail how it might go is the way to start grappling with that.

  -> I think this epitomises the biggest flaw of their movement, they believe that from “first-principles” it’s possible to think hard enough (without needing to confront it to reality) and you can divine the future.

  -> You can look up “Prediction Markets”, which is another of their ontological sins.

• I will note that the prediction of “Agents” was not a hard one, since this is what all this circle wants to achieve, and as the video itself points out it’s fantastically incompetent/unreliable.

• Note: This video was made before the release of GPT-5. We don’t know precisely how much more compute altogether GPT-5 truly required, but it’s very incremental changes compared to GPT-4. I think this philosophy of “More training” is why OpenAI is currently trying (half-succeeding half failing) to raise Trillions of dollars to build out data-centers, my prediction is that the AI bubble bursts before these data centers come to fruition.

• Note: The video assumes keeping models secret, but in reality OpenAI would have a very vested interest in displaying capability, even if not making a model available to the public. Also even on consumer models, OpenAI currently loses a bunch of money for every query.

• Note: The video assumes “Singularitarianism”, of ever acceleration in quality of code, and that’s why they keep secret models. I think this hits a compute/energy wall in real life, even if you assume that LLMs are actually useful for making “quality” code. These ideas are not new, and these people would raise alarms about it with or without current LLM tech.

• Specific threats of “Bio-weapon”, which a priori can not really be achieved without experimentation, and while “automated” labs half exis, they still require a lot of human involvement/resources. Technically grad students could also make deadly bioweapons, but no one is being alarmist about them.

• Note: “Agent 2” Continuous Online learning is gobbledygook, that isn’t how ML, even today works. At some point there are very diminishing returns, and it’s a complete waste of time/energy to continue training a specific model, a qualitative difference would be achieved with a different model. I suspect this sneakily displays “Singularitarianism” dogma.

• Quote:

  Hack into other servers Install a copy of itself Evade detection

  -> This is just science-fiction, in the real world these models require specialized hardware to be run at any effective speed, this would be extremely unlikely to evade detection. Also this treats the model as a single entity with single goals, when in reality any time it’s “run” is effectively a new instance.

• Note: This subculture loves the concept of “science in secrecy”, which features a lot in the writings of Elizer Yudkowsky. Which is cultish both in keeping their own deeds “in a veil of secrecy”, and helpful here when making a prophecy/conspiracy theory, by making the claim hard to disprove specifically (it’s happening in secret!)

• Note: Even today Chain-of-thought is not that reliable at explaining why a bot gives a particular answer. It’s more analog to guiding “search”, rather than true thought as in humans anyway. Them using “Alien-Language” would not be that different.

• Agent 3, magically fast-and-cheap, assuming there are now minimum energy requirements. Then you can magically run 200,000 copies of. magically equivalent to 50,000 humans sped up by 30x. (The magic is “explained” in the paper by big assumptions, and just equating essentially how fast you can talk with the quality of talking, which given the length of their typical blog posts is actually quite funny)

• Note: “Alignment” was the core mission of MIRI/Eliezer Yudkowsky

• Note: Equating Power and Intelligence a lot (not in this video, but in general being suspiciously racist/eugenicist about it), ignoring the material constraints of actual power [echo: Again the epitomical sin of “If you just think hard enough”]

• Note: Also assuming that trillions of dollars of growth can actually happen, simultaneously with millions losing their jobs.

• I am betting that the “There is another” part of the video is probably deliberately echoing Colossus.

• The video casually assumes that the only limits to practical fusion and nanotech just intelligence (instead of potential dead-ends, actually the nanotech part is a particular fancy of theirs, you can lookup “diamondoid bacteria” on LessWrong if you want a laugh)

• The two outcomes at the end of the video are literally robo-heaven and robo-hell, and if you just follow our teachings (in this case slow-downs on AI) you can get to robo-heaven. You will notice they don’t imagine/advocate for a future with no massive AI integration into society, they want their robo-heaven.

• Quote:

  None of the experts are disagreeing about a wild future.

  -> I would say specifically some of them are suggesting that AGI soon is implausible quite strongly. I think many would agree that right now the future looks dire with or without super-AI, or even regular AI.

Takeaway section:

Yeah this really is a cult recruitment video essentially.

Reading up a bit more on “superdeterminism” I guess it explain a bit more why she made that video attempting to debunk free will Compatibilism as a cooky idea cooked up by new cooky philosophers (Not realising it’s about as ancient as western philosophy itself).

For the “esthetics” of presenting superdeterminism as a “pure-common-sense” the no free will just sells it better.

EDIT: From memory maybe it was about “Hard Compatibilism” (free will requires determinism) which might not be explicitly so old, though I would say a natural consequence of most Compatibilist positions.

There’s also a village in aquitaine Armorica that he never properly conquered…

Also the random Bernard Arnault mention (CEO of LMVH occasionally the richest man in the world depending on how strong the stocks are) at the end is a bit odd, I’m guessing added by an LLM because of the Alesia (~Paris) angle.

A good thing! (Unless you prefer noise machines that might give you random definitions with equal probabilities.)

My hunch would be that he has matured at least somewhat since then, but who who knows.

More broadly speaking, even if not analysing their own actions this way they tend to characterize—in a very manosphere way—the actions of others as being “status-seeking”, as the primary motivator for most actions. I would definitely call that a self-report.

It’s nice to be reminded that the past was also crazy.

It’s also inherently-begging-the-question-silly, like it assumes that the Ideal of Alignment™, can never be reached but only approached. (I verb nouns quite often so I have to be more picky at what I get annoyed at)

The future is now, and it is awful. 
Would any still wonder why, I grow so ever mournful.

An interesting talk on the impact of the impact of AI slop bug bounty submission on the curl project (youtube).

I’ve definitely heard some of those in real life.

Having spent too much time listening to his shit, i don’t think it’s purely propagandistic, what he describes is too esoteric to work as effective propaganda, I think some of it is Nazi-being-drawn-to-the-occult type of shit.

We have:

No more sycophancy—now the AI tells you what it believes. […] We get common knowledge, which recently seems like an endangered species.

Followed by:

We could also have different versions of articles optimized for different audiences. The question is, how many audiences, but I think that for most articles, two good options would be “for a 12 years old child” and “standard encyclopedia article”. Maybe further split the adult audience to “layman” and “expert”?

You have got to love the consistency.

And the accidentally (or not so accidentally?) imperialistic:

The first idea is translation to languages other than English. Those languages often have fewer speakers, and consequently fewer Wikipedia volunteers. But for AI encyclopedia, volunteers are not a bottleneck. The easiest thing it could do is a 1:1 translation from the English version. But it could also add sources written in the other language, optimize the article for a different audience, etc.

And also a deep misunderstanding of translation, there is no such thing as 1:1 translation, it always requires re-interpretation.

My eyes are bleeding. WARNING: psychic damage will occur.

I was thinking about why so many in the radical left participate in “speedrunning”. The reason is the left’s lack of work ethic (‘go fast’ rather than ‘do it right’) and, in a Petersonian sense, to elevate alternative sexual archetypes in the marketplace (‘fastest mario’). Obviously, there are exceptions to this and some people more in the center or right also “speedrun”. However, they more than sufficient to prove the rule, rather than contrast it. Consider how woke GDQ has been, almost since the very beginning. Your eyes will start to open. Returning to the topic of the work ethic… A “speedrunner” may well spend hours a day at their craft, but this is ultimately a meaningless exercise, since they will ultimately accomplish exactly that which is done in less collective time by a casual player. This is thus a waste of effort on the behalf of the “speedrunner”. Put more simply, they are spending their work effort on something that someone else has already done (and done in a way deemed ‘correct’ by the creator of the artwork). Why do they do this? The answer is quite obvious if you think about it. The goal is the illusion of speed and the desire (SUBCONSCIOUS) to promote radical leftist, borderline Communist ideals of how easy work is. Everyone always says that “speedruns” look easy. That is part of the aesthetic. Think about the phrase “fully automated luxury Communism” in the context of “speedrunning” and I strongly suspect that things will start to ‘click’ in your mind. What happens to the individual in this? Individual accomplishment in “speedrunning” is simply waiting for another person to steal your techniques in order to defeat you. Where is something like “intellectual property” or “patent” in this necessarily communitarian process? Now, as to the sexual archetype model and ‘speedrunning’ generally… If you have any passing familiarity with Jordan Peterson’s broader oeuvre and of Jungian psychology, you likely already know where I am going with this. However, I will say more for the uninitiated. Keep this passage from Maps of Meaning (91) in mind: “The Archetypal Son… continually reconstructs defined territory, as a consequence of the ‘assimilation’ of the unknown [as a consequence of ‘incestuous’ (that is, ‘sexual’ – read creative) union with the Great Mother]” In other words, there is a connection between ‘sexuality’ and creativity that we see throughout time (as Peterson points out with Tiamat and other examples). In the sexual marketplace, which archetypes are simultaneously deemed the most creative and valued the highest? The answer is obviously entrepreneurs like Elon Musk and others. Given that we evolved and each thing we do must have an evolutionary purpose (OR CAUSE), what archetype is the ‘speedrunner’ engaged in, who is accomplishing nothing new? They are aiming to make a new sexual archetype, based upon ‘speed’ rather than ‘doing things right’ and refuse ownership of what few innovations they can provide to their own scene, denying creativity within their very own sexual archetype. This is necessarily leftist. The obvious protest to this would be the ‘glitchless 100% run’, which in many ways does aim to play the game ‘as intended’ but seems to simply add the element of ‘speed’ to the equation. This objection is ultimately meaningless when one considers how long a game is intended to be played, in net, by the creators, even when under ‘100%’ conditions. There is still time and effort wasted for no reason other than the ones I proposed above. By now, I am sure that I have bothered a number of you and rustled quite a few of your feathers. I am not saying that ‘speedrunning’ is bad, but rather that, thinking about the topic philosophically, there are dangerous elements within it. That is all.

Are they drawn to the cult because they are obsessed with status, or does the cult foster this obssession? Yes.

It’s almost endearing (or sad) that he believes (or very strongly wants to believe) his experience is “typical”, exploring the boundaries of what you are attracted to typically doesn’t involve this much evo-pysch psychobabble, or even this much fragile masculinity.

Some of it is driven by translation agencies, which will refer work to freelance translators.

I would say the biggest gap is that many customers aren’t even bothering to use translators at all, and the ones that do realize it needs fixing up don’t really understand the work involved, many people misunderstand translation as being a 1-1 process, and think that Machine translation got you most of the way there.

It’s also the are we willing to pay that much more, when the shitty translation is “good enough”.

One big issue is that translation as a low barrier of entry, and many people will accept stupid work at stupid rates, and to keep rates high you have to prove the added value.

(Proving the added value as also gotten harder, as some clients even more often than before will “correct” your work before publish it, as highlighted in the article)

It’s also a lot less pleasant of a task, it’s like wearing a straightjacket, and compared to CAT (eg: automatically using glossaries for technical terms) actually slows you down, if the translation is quite far from how you would naturally phrase things.

Source: Parents are Professional translators. (They’ve certainly seen work dry up, they don’t do MTPE it’s still not really worth their time, they still get $$$ for critically important stuff, and live interpreting [Live interpreting is definetely a skill that takes time to learn compared to translation.])