The major selling point for these LLM services is that people don’t need to learn a strict formal query language to interact; the LLM consumes instructions and data all as an undistinguished stream, and statistically infers a plausible response with absolutely no regard to the input’s meaning.

That is: despite the media reporting these as “injection attacks”, that term means nothing when all its input is treated as data and instruction simultaneously. This isn’t some special class of attack; it’s a fundamental designed-in flaw of the system.

The correct way to ensure protection from these vulnerabilities, is to never treat the input data as instructions, but instead to have a separate channel for query instructions and to reject bad input on that channel.

But of course that would kill the major appeal for most people who love these things, the fact they don’t need to learn any strict formal language and can just say anything at all and get some useful-looking response. Take that away, and you lose any hope that the masses will want to use these systems.

And so the makers and promoters of these systems will never make the one change that could even feasibly allow safety from these attacks; they will never make any improvement to security that might reduce the apparent ease of use of these things.

For as long as that remains, these systems will continue to inevitably have these exploits, and there’s no way to close the exploit surface. These are staggeringly insecure by design, and can’t be fixed without being completely replaced.

David G: “What happens after the bubble pops? What do we do amongst the wreckage of what was once the AI bubble?”

Tante: “A question that really concerns me is [the data centres that have been built]; who will they [the future owners of data centres, after the hyperscalers stop paying] turn to, to get their money back? The only entities that I can think of that have the money to make this worth it, is the military, and the security apparatus. […]

“That is useful for a lot of shit that we don’t want. That is a problem that really concerns me. Who has problems that fit to pattern-matching machines; and who has the money [to pay data centres to address those problems]? And that’s gonna be a lot of military applications, that’s gonna be a lot of security apparatus applications, that we all probably should not want.”

David G: “That’s wonderfully probable and depressing.”

Tante: “I’m fun at parties.”

The end result will probably be some sort of copy-protection system which will benefit those who want to censor the web and create paywalls and other gatekeeping methods to block access to content. We’re better off with the web as it is now, written entirely in plaintext. Besides, obfuscation goes against the spirit the World Wide Web was founded on: free and open access to information for the benefit of humanity.

Emphatically agreed. The open web was already being closed down before the LLM hyperscalers; it is closing down even more rapidly, partly to defend against the LLM hyperscalers dunning every website with pointless scraping all the time.

If we respond by obfuscating the text on the web, we hurt ourselves enormously and still won’t stop the LLM hyperscalers from ruining the web. It’s a misguided, technical non-solution, for something that needs a political response.

AI systems are never going to start getting dumber.

…hasn’t this already happened many times???

I mean, I guess it can be seen as technically true, because LLMs (the subset of AI they’re clearly talking about here) already know exactly zero about reality or what’s true. Can’t know less than that.

finding the [human created] data to support the project is uncharted territory, he added.

Fortunately for the executives at hyperscaler corporations, they know this isn’t their problem.

They plan to cash out well before the model collapse is revealed, and leave less-wealthy suckers holding the bag.

It’s like killing the goose that lays the golden eggs.

Goose was not valued

I have no idea why they would do this

I do. For the same reason they are cramming LLMs into every other product:

They need to pretend to their investors that there is an enormous market hungry for more of this, so the investors will not demand their money back.

By turning more users of existing services, into “users of Gemini”; by turning more translate requests, more email messages, more search requests, into a “Gemini usage”; they can with a straight face claim more and ever more usage and demand, propped up by users who never agreed to any of this.

From TFA:

So what do we search with instead of Google? There isn’t a lot of choice. There’s various flavours of Google or Bing.

[…] Kagi resells Google’s actual link results for a monthly subscription. Kagi is really an AI company that sells search as a sideline, and they’ve got a terrible AI slop news service too. But the search has a good reputation for now.

GEMA versus OpenAI. Whoever wins, we all lose.

But I’m still happy to see them fight.