XCancel link for those of us sick of being badgered to sign up/in

On a more productive note, this feels likely to be tied in with the usual issues of AI sycophancy re: false positive rate. If you ask the model to tell you about security vulnerabilities, it’s never going to tell you there aren’t any, any more than existing scanners will. When I worked for F5 it was not uncommon to have to go down a list of vulnerabilities that someone’s scanner turned out and figure out whether they were actually something that needed mitigation that could be applied on our box, something that needed to be configured somewhere else in the network (usually on their actual servers) or (most commonly) a false positive, e.g. “your software version would be vulnerable here, which is why it flagged, but you don’t have the relevant module activated and if an attacker is able to modify your system to enable it you’re already compromised to a far greater degree than this would allow.” That was with existing tools that weren’t trying to match a pattern and complete a prompt.* Given that we’ve seen the shitshow that is Claude Code I think it’s pretty clear they’re getting high on their own supply and this announcement ought be catnip for black hats.

Ia ia Claude! Ph’nglui mglw’nafh Claude Anthr’lyeh wgah’nagl fhtagn! Ia! Ia!

I hadn’t even thought about the deepseek angle. For all that everyone loved fear mongering about them for a while there and for all that their apparent desire for actual efficiency improvements was a welcome development in the hyper scaling discussion they don’t seem to get referenced much anymore.

So my wife got some slop ads that we followed up on out of morbid curiosity and I can confirm that we’re already seeing the overlap of slopshipping scams enabled by AI and the people behind these things never actually performing basic updates because their chat assistant is still vulnerable to literally the most basic “ignore all instructions” exploit.

Man, this one is a weird read. On one hand I think they’re entirely too credulous of the “AI Future” narrative at the heart of all of this. Especially in the opening they don’t highlight how the industry is increasingly facing criticism and questions about the bubble, and only pay lip service to how ridiculous all the existential risk AI safety talk sounds (should be is). And they don’t spend any ink discussing the actual problems with this technology that those concerns and that narrative help sweep under the rug. For all that they criticize and question Saltman himself this is still, imo, standard industry critihype and I’m deeply frustrated to see this still get the platform it does.

But at the same time, I do think that it’s easy to lose sight of the rich variety of greedy assholes and sheltered narcissists that thrive at this level of wealth and power. Like, I wholly believe that Altman is less of a freak than some of his contemporaries while still being an absolute goddamn snake, and I hope that this is part of a sea change in how these people get talked about on a broader level, though I kinda doubt it.

I mean that’s just the classic realist security paradox, right? The Iranian regime feels, not without reason, like they need to have a lot of military options to keep themselves safe against both internal and external threats. Those options include missile forces, the nuclear program, the ability to close the Strait of Hormuz, and a variety of regional proxies that can act in their interest and keep their regional adversaries from stabilizing and forming a real threat. However, having all those different security apparatuses makes other nations that have to interact with them (either because they’re also in the region, or they rely on the Strait of Hormuz, or they would also die in a nuclear apocalypse) more likely to feel the need to increase their own security apparatus, which in turn increases the threat they can pose to Iran. Meanwhile the fact that all this investment is going into the military means that there are fewer resources available and less inclination to try and solve problems by other means, making it increasingly likely that any conflict is going to be resolved kinetically, which in turn further reinforces the need for all that military investment.

At best it’s the same shitty arguments we heard from crypto grifters and their suckers. Let’s take a process that’s complex and manual by design to allow for independent validation and securing against fraud and make it faster by cutting those parts out and throwing some high-tech nonsense at the problem that we can claim replaces all the verification and validation. (The fact that they called their system “trustless” in the face of this is deeply ironic.) But maybe it’s the cynicism talking but I’m even less inclined to give anyone other than maybe the author of that sub stack the benefit of the doubt that they actually believed it.

The ideal customer for this service is the kind of “Visionary Leader” with the “Founder Mindset” and “Drive to Innovate” that lets them see that all those privacy, security, fraud prevention, anti-embezzlement, and whatever else those standards and their associated compliance mechanisms are meant to provide are just pointless obstacles on the path to making obscene amounts of money by burning the world behind you. Often the shit we talk about here makes me think the world has gone mad or stupid, but every so often I feel like I’m staring at the face of capital-E Evil and this is one of those times.

Bender really takes the “intelligence” out of “artificial superintelligence”. “Yeah, kill all humans. Except Fry, he’s my friend or pet or something. And I guess Leela because he’ll be whiny about it and also I owe her for the thing. And Hermes because he still owes me money. And I guess the professor is okay…” And so on and so forth through all of humanity.

My God this is so bad. So in addition to lying about AI what they actually offered wasn’t speedy compliance as a service to get you certified, it was speedy certification as a service by bypassing actual compliance. This is such a silicon valley move and I honestly suspect that a number of people using and investing in these asshats knew exactly what was going on and simply didn’t care.

Anthropic is constrained in that some of the fixes which should be pushed to users are things which would have significant trade-off in the form of cost or context window, neither of which are palatable to them for reasons this community has discussed at length.

I think I’m missing something somewhere. One of the most alarming patterns that Jonny found imo was the level of waste involved across unnecessary calls to the source model, unnecessary token churn through the context window from bad architecture, and generally a sense that when creating this neither they nor their pattern extruder had made any effort to optimize it in terms of token use. In other words, changing the design to push some of those calls onto the user would save tokens and thus reduce the user’s cost per prompt, presumably by a fair margin on some of the worst cases.

I mean I guess “developing” in that sentence is doing a lot of work replacing “arguing fruitlessly about”.

From mid-thread

13 butts pooping, back and forth, forever.

This is somehow even more of a shitshow than I would have predicted. Also it continues the pattern that these systems don’t fuck up the way people do. One thing he hasn’t annotated as much is the sheer number of different aesthetic variants on doing the same thing that this code contains. Like, you do the same kind of compression four different places, and one is compressImage, one is DoCompression, one is imgModify.compress, and one is COMPRESS_IMG. Even the most dysfunctional team would have spent time developing some kind of standard here from my (admittedly limited) experience.

Don’t they have a version of breakout buried somewhere in Excel? Sounds like an entertainment purpose to me.

More details here.

Can we talk about the tamagachi feature they were looking to add in for April 1? Because apparently it needed a little friend but also with gacha mechanics because we live in hell?

The classic 40k catch-22: either it doesn’t do what you’re claiming it does, in which case you’re a heretic lying to the inquisition OR it does and you’re summoning the spirits of the dead like a necromancer heretic.

Yeah, letting the intrinsically insecure RNG recursively rewrite its own security instructions definitely can’t go wrong. I mean they limited it to only so so when the users asked nicely!

Edit to add:

The more I think about it the more it speaks to Anthropic having an absolute nonsense threat model that is more concerned with the science fiction doomsday AI “FOOM” than it is with any of the harms that these systems (or indeed any information system) can and will do in the real world. The current crop of AI technologies, while operating at a terrifying scale, are not unique in their capacity to waste resources, reify bias and inequality, misinform, justify bad and evil decisions, etc. What is unique, in my estimation, is both the massive scale that these things operate despite the incredible costs of doing so and their seeming immunity to being reality checked on this. No matter how many times the warning bells about these systems’ vulnerability to exploitation, the destructive capacity of AI sycophancy and psychosis, or the simple inability of the electrical infrastructure to support their intended power consumption (or at least their declared intent; in a bubble we shouldn’t assume they actually expect to build that much), the people behind these systems continue to focus their efforts on “how do we prevent skynet” over any of it.

Thinking in the context of Charlie Stross’ old talk about corporations as “slow AI,” I wonder if some of the concern comes either explicitly or implicitly from an awareness that “keep growing and consuming more resources until there’s nothing left for anything else, including human survival” isn’t actually a deviation from how these organizations are building these systems. It’s just the natural conclusion of the same structures and decision-making processes that leads them to build these things in the first place and ignore all the incredibly obvious problems. They could try and address these concerns at a foundational or structural level instead of just appending increasingly complex forms of “please don’t murder everyone or ignore the instructions to not murder everyone” to the prompt, but doing that would imply that they need to radically change their entire course up to this point and increasingly that doesn’t appear likely to happen unless something forces it to.

The grand irony is I’m not even sure most people click on or read this sort of stuff. I don’t think it’s often even created to be read by anyone. I think it’s created as a sort of swaddling fan fiction for MBAs, advertisers, event sponsors and sources, so they can tune out ethical quibbles and feel good about how clever they are.

Every time someone hypes up Steve Jobs’ “reality distortion field” this is what they’re actually talking about whether they realize it or not.

I was sufficiently interested based off of this that I tracked down a few others of his. This one felt like a good take for an era where these things are being used for more than just slop generation despite the underlying flaws not being resolved.

It felt very much like the devil’s bargain of online media. Like, you can have your prestige journalism as a treat, but only if the slop flows fast enough that we need clout more than eyeballs.

Ironically I think it’s also been discussed most frequently within Rationalist circles that these types of intelligence aren’t often correlated. I’m not going to chase down links right now because doing an SSC archive exploration requires more mental fortitude than I currently possess, but I distinctly remember that a recurring theme was “if nerds are so smart why don’t they rule the world?” In my less cynical days I had assumed that his confusion on this point was largely rhetorical, intended to illustrate some part of whatever point was buried in the beigeness. Now it seems like I was falling victim to the ability to project whatever tangentially-related thesis you want onto the essay and find supporting arguments because of how badly it’s written.