Anthropic: AI MIGHT KILL US ALL WITH A BIOWEAPON

Also Anthropic: starts an AI-run bio lab. If it’s run to the same standards as the cybersecurity division I can’t imagine it’s going to be particularly well-supervised.

Do you want to get T-Virus? Because this feels like how you get T-Virus.

How would you even go about turning a huge bar of gold into cash anyways?

It might be wishful thinking, but it would also answer the question: why now? It’s not like the concerns are meaningfully different now compared to when the bubble started. If they were so convinced that the AI future was both inevitable and possibly catastrophic, they could have followed Yud into the “research foundation” game rather than accelerating the timeline. I guess they’re probably more convinced than I am that the latest round of security breaches are actually a sign of new capabilities in the models, but it still feels really convenient. Frontier model advancements have been slowing down and costs have been accelerating exponentially, and now they’re taking the line that they’re choosing to slow progress and spend less money on arguably their biggest cost center.

Our AI cult keeps doing whatever they think the AI says and we can’t make them stop.

I’m stealing EBBT for these conversations. Much easier to follow than “TNR (technically not revenue)”.

I actually love the pigeon missile analogy here. Sure, you can wax poetic about how impressive it is to train a pigeon to do much of anything, and you did have to work hard and apply specialized knowledge and whatever to make it happen, but outside of animal training circles the pigeon is the least interesting or dangerous part of the whole contraption.

I think the FOSS angle kinda works just off of the inherently unsustainable economics of the space. Relying on unpaid labor makes the worker protection arguments inherently less valuable because it’s not like this was your livelihood to begin with (ostensibly). Keeping up with the next thing is a pretty clear way to keep a mostly-volunteer workforce interested.

I mean, I guess the consequences and policies associated with culture war shit did abruptly go from ridiculous to real fucking serious what with the camps and the trying to unperson trans people and the overturning Roe v Wade. Maybe that leaves the Dems (who I assume are still being treated as “the left” in this discussion) in a more ridiculous position?

Not using :wq! because consent is important

I feel like given the usual experience of uninitiated players with that coffin we need to acknowledge the bearer of the curse as canonically trans.

I was going to try and sneer this further but I realized that my own narrative of political polarization almost started with Clinton and I’m in my early 30s, meaning that the problem is obviously whoever the president was when the person making the argument was a child.

I agree that this is more plausible, but it’s still almost as baffling, since it suggests that the existence of some people who can afford a (hypothetical) bed-making robot implies the nonexistence of people who can’t. Like, at least Ovacado toast in a cafe was a vaguely affordable luxury that someone could conceivably stretch their means to occasionally purchase. This is more like “all you need to do to stop being poor is cut back on having your chauffer paint himself in gold leaf to match your limo”.

The general idea that “we asked the robot to draw a scary face and then got scared by the face it drew” is still relevant, but the details increasingly matter here. They may not be actually thinking but they are increasingly being allowed to act autonomously and with wildly insufficient oversight.

I need someone with more actual software dev knowledge to back me up on something. It looks like there’s a new zero-interaction remote code execution vulnerability in most major vibe code generators. I’ve seen it called Plugin4Shell because apparently we’re still doing marketing names instead of actually getting CVE numbers for these things. The link there seems like a decent overview.

The bug, dubbed Plugin4Shell, breaks SHA pinning, the mechanism developers rely on to lock an installed plugin to a specific, reviewed version of its code. Pinning is supposed to mean that once a plugin passes review, it cannot change without the developer’s knowledge.

AIR’s researchers found that every one of the four agents checks out the pinned commit without verifying the checkout landed there, letting an attacker swap in malicious code while the pin still looks intact.

So if I’m reading this right, the mechanism to cryptographically ensure that your AI agents are using the code they say they are just was straight-up not being checked? This feels like it should go on the big board of incredibly obvious failures, but I’m not familiar enough with the git side of things to be absolutely confident in that. Like, if a person tried to pass off software that did this it would have significant career implications, right? Or am I misunderstanding something somewhere?

It feels like they’re trying also to privilege AI as “system 3” when realistically the idea of external cognitive assistance isn’t that different if you’re looking at Claude the chatbot or Claude the smart kid in your class who lets you copy his homework before homeroom. You could probably distinguish this from other forms of paratext by the degree to which it seeks to actively replace your own thought rather than supplementing or augmenting it, but then I remember my teachers saying the same thing about sparknotes so there’s probably more of a continuum there. If that’s the case, then the pattern that Dan Olsen describes in his WoW documentary is probably relevant here as well, since it’s performance pressure and social norms that are really pushing the negative practices into a level that is killing everyone’s souls.

I think cognitive offloading has come up a number of times here, but I don’t know that cognitive surrender has been brought up as much. It reminds me of The Colonization of Confidence by Robert Kingett, particularly the stretch where Leo is convinced that his actual voice as a writer is strictly inferior to the generic AI slop. There’s a sense of active despair there that seems worth acknowledging if we’re going to have that conversation.

I thought learned helplessness was tied more to acceptance of negative stimuli. Like, if you expose a rat to an electric shock that it can’t do anything to escape or prevent enough times they’ll learn that they can’t do anything about it and stop trying even if presented with a very clear means of escape. Cognitive offloading I think is more tied to atrophy and/or habituation, which is related but distinct imo.

10 PRINT "I AM ALIVE"/r/n 20 GOTO 10

AI says it’s alive, please give me a billion dollars now.

I feel like the problem is less one of infiltration and more one of context drift. The racism and eugenics were in there, but it wasn’t as central to the group as the fun sci-fi stuff so people could pretend it wasn’t there, the same way that people pretend not to know that their coworker heretically believes in the wrong pizza toppings or whatever. Effectively they’re saying “Sure, they may have some bad takes on race or gender or economics but they’re the only ones who will talk about the second coming of Robot Jesus with me.” This dynamic also largely explains why these groups (and arguably sci-fi in general?) are so goddamn white and male.

Of course, as the tech industry took off and these people sought and acquired actual wealth and influence, the dynamic started to reverse. The fascists may not be appropriately invested in the second coming of Robot Jesus, but they are in agreement about eugenics and deregulation, and that’s enough for the tech oligarchs to throw their now-considerable resources behind them.

I usually have some sympathy for the dev/maintainer side of FOSS since it is very real work and passion doesn’t exactly pay the bills, but the proliferation of AI nonsense is deeply disheartening. At best it shows how narrow of a band these people select from, since even among programmers and software nerds, pro-AI sentiment seems tied to a specific subgroup. Less charitably it shows how the FOSS movement has been fully captured by the corporate interests who always saw it as a source of free labor first.