it just has that feel.

yeah. Looking past the initial quiz, a lot of the complaints about Eliezer’s predictions feel like rationalist inside-baseball about takeoff speeds and alignment strategies, as opposed to calling the whole thing bunk.

Calculators also have a negligible chance of screwing up something basic, and you can in fact learn the right way to input problems into them such that they are guaranteed not to misinterpret you.

Wow… I actually failed to get some of the times Eliezer has been wrong! His predictions are even worse than I remembered!

Thanks! I had failed to check back two stubstacks ago.

In short, here, I described this not as collusion

other similar systems

Something of a side note to your excellent write-up… I think we should push back on the “sub-agent” “agent swarm” language used to describe LLM workflows.

The workflows often described as swarms or whatever are the same LLM (or at least related LLMs), just prompted a bunch of different ways in parallel. Individual agents don’t really act like coherent agents in the lesswrong rationalist sense, or in any sort of philosophical sense of selfhood or agency or unified purposes, they just get described that way for convenience, so treating “swarms” of agents and subagents as some special category is lending too much credence to the anthropomorphisizing boosters like to do. “Agent swarms” really just means that someone set up a slop machine to prompt itself a bunch of times in parallel without enough human supervision.

A lesswronger correctly notices that LLM companies’ “research” papers are often lacking in details necessary to replicate them! They fail to notice the reasons why that is, or do any broader questioning or soul searching about the state of LLM and “AI Safety” “research”.

This has been talked about in the reddit sneerclub, but not yet on awful systems (as far as I could tell with a quick search on this and last week’s stubstack)…

Caroline Ellison (the convicted FTX fraudster) got hired for an EA position dealing with financial allocations!

Reddit sneerclub discussion

EA Forums Discussion surprisingly (by the low low bar we have for EA) this announcement was heavily downvoted and all the top comments clearly and frankly denounce Manifund for hiring Ellison, accurately calling out the sheer violation of common sense and the obviously unethical nature of hiring someone because they previously directed money your way that turned out to be stolen. (The Manifund person outright admits this as a motivation!) Not so surprisingly, less upvoted, but still upvoted comments do some hand-wringing about the optics (as opposed to the clear ethical and common sense issues) and some even outright call for forgiveness for Caroline!

The lesswrong discussion is less surprising (as in it is full classic lesswrong fallacies). It starts with considering the optics, and some posters, such as habryka, outright question the existence of any such ethical rule that hiring Caroline might violate! (If there isn’t a rule anywhere, its only because “don’t hire convicted fraudsters for positions dealing with a lot of money” is such blatant common sense.)

Bonus: A galaxy brain take: maybe since Caroline has already messed up so massively, that means she actually has the best experience!

True… except for the fact that some of them were there at the start. Like, for example, Nick Bostrom had some pretty racist/eugenicist stuff come out from the 90s when he was on some mailing lists and 90s nets. Nick Bostrom was a founding member of the Future of Humanity Institute, along with Toby Ord (another EA founding member) and they got money from Elon Musk.

What choice did they have. The media and the masses ignored the clear bio threat from AI (as undeniably proved by them getting a 99% on biology focused multiple choice quizzes rigorous PhD level benchmarks, which everyone knows is a great way to evaluate PhD level biology and wetlab skill). And the media took the hugging faces hack seriously. So clearly Anthropic just needs to show their AIs manufacturing a genuine pandemic!

HPMOR does have one great lesson that all the rationalist and even Eliezer himself have failed to internalize: Fascists should be decapitated.

The EA/lesswronger attitude of “assuming good faith” (to the right people) and taking shit too seriously does play perfectly off of Trump’s blatantly hollow and shallow rhetorical shenanigans.

A year might be enough time for the bubble to start leaking faster or the pop to become outright undeniable. And from what I’ve picked up from Ed Zitron’s reporting, a lot of the bigger data center deals are kind of tenuous to begin with (sketchy financing, no ability to actually meet overly aggressive timelines even with optimal money and no regulation, etc.). So anything helps.

Going on a bit of a tangent… Eliezer (and lots of rationalists) are so certain only international deals driven by their ideology have any hope of really stopping AGI (Eliezer’s Only Law Can Prevent Extinction is mostly about not supporting direct extralegal action against AI, but it also implies this, and of course AI 2040 is fixated on some cartoonishly impossible international deal), it would be funny if the surge data center opposition for purely pragmatic reasons prove to be a major cause of the bubble finally popping.

See, if they just came out and admitted to that sort of strategy, I could at least respect it some, but instead they still seem to insist on acting like and planning for Trump being somewhat of a rational actor!

I’ve previously made an effort post summarizing this! https://awful.systems/post/4172436

tldr; Eliezer has criticized (more like mocked, or dare I say it, sneered at) expert systems, GOFAI, and neural networks, and he never really admitted he was wrong when it became apparent in the 2010s that neural networks really were a viable approach.

The exchange with niknoble in the comments was also my favorite. He notes the democrat party is actually the best bet for getting regulation on AI:

More generally, acceleration wins by default if everyone follows their local financial incentives, and the only force I’ve seen overpower capital at scale is the will of the Democratic Party of the United States.

He is maybe giving Democrat too much credit, but yeah, if the doomers were really trying to win, they should be looking at the party that at least has a chance of agreeing to more regulations.

It annoys me how much credit for predicting AI Eliezer gets away with claiming among his cult (and even outside his cult, like in interviews with reporters that could know better if they bothered to do their job) when all the actual details he predicted are nothing like LLM-based agents.

Likewise for claiming success on predicting instrumental convergence. LLM agents often can’t stay focused on the goal they are post trained and directly prompted to pursue, they are the furthest thing from trying to optimally maximize a bunch of secondary values in pursuit of master goal.

You know… from a big brain game theory perspective (rationalists love trying to take that perspective)… Eliezer is actually rewarding Trump’s tantrum by demanding EA bow to the right.

Rationalist have never run into a right-wing point or person they were willing to steelman nor a leftist they were unable to screw up steelmanning.

It would be hilarious if the AI doomers actually did manage to get Trump to sign some of their ideas into action with executive orders, only for him to reverse course or fail in implementation details, and then a hard tide of anti-Trump sentiment in the next administration to totally undo them.

Sadly it looks like we will only get the more immediate humor of Trump going hard against them.

Lol, I called it without even having to see it. Doubling down and blaming EAs for being too left even as the right curbstomps them. How does that boot taste?

Also, WTF does Eliezer even think “leftist” means?