• 0 Posts
  • 205 Comments
Joined 2 years ago
Cake day: May 20th, 2024
  • BioMan@awful.systemstoTechTakes@awful.systems•Stubsack: weekly thread for sneers not worth an entire post, week ending 15th November 2026English
    0·
    3 hours ago

    These scaling laws are even funnier than that, as are more recent ones.

    I continue to be extremely amused by the fact that you can take a LLM ‘reasoning’ trace, replace all the ‘reasoning’ with dashes or spaces, re-run the attention mechanism, and have what comes out the other end improve by a significant fraction as much as if you let it exude text that has the appearance of having to do with the topic at hand. It’s just more opportunity for the embedding vectors to dance against each other before you translate the abstract vectors back into natural language. Hence recent frontier models doing a thing where instead of doing a long reasoning trace, they build the attention matrix, allow the vectors to push each other around… then let those new vectors push each other around again, and again, and again, multiple rounds, before you start decoding.

    More recently, I have seen some vague math into the scaling of the speed with which running many parallel systems with access to a common pool of text produce a legible result. If I recall, it comes to roughly:

    N parallel systems

    Finish in ~1/N^0.5 as much time

    At ~N^0.5x the cost

    So 16 parallel systems crosstalking finish complex tasks in 1/4 the time at 4x the cost (and 16x the instantaneous rate of operation per second)

  • BioMan@awful.systemstoTechTakes@awful.systems•Stubsack: weekly thread for sneers not worth an entire post, week ending 8th November 2026English
    0·
    10 days ago

    Oh my god

    So why, at Proton, does it not feel like the revolution has come? Why has productivity not increased dramatically if everyone is using LLMs all the time already?

    The first is a change in engineering culture and engineer mentality. Your job has fundamentally changed. It is no longer to write code, and LLMs are not here to help you write better code faster. Your job is to help the LLM write better code faster.

    The second reason we have not been able to take full advantage of the AI revolution is that we have been almost completely stuck at phase 1 — using LLMs to speed up tasks as part of existing workflows. But writing code is not the only part of the software development lifecycle, and what happens when you eliminate a bottleneck? Another takes it is place.

    The third reason is a collective action problem. Using LLMs effectively to write code is something an individual engineer can do for themselves… But to retool the company at large requires amounts of both organization and resourcing that we simply have not invested in yet.

  • BioMan@awful.systemstoTechTakes@awful.systems•Stubsack: weekly thread for sneers not worth an entire post, week ending 1st November 2026English
    0·
    10 days ago

    You can have a system that just gives you a relevance-sorted list, or a system that then ingests the chunks of those documents that are most relevant and gives a human readable summary in addition to the list.

    The indexing step requires a lot of processing, by which you chunk reference documents into couple hundred word chunks and cook each of those chunks into a high dimensional embedding vector. Once you have done that and saved the database of vectors, you can put in your query, embed it, check its dot product against all the other embedding vectors, and get your ranked list (potentially quite fast) without need to produce text out the other end.

    Making those embedding vectors works much faster on a GPU rather than a CPU, but can be done on either. Since I have one system set up to ingest a bunch of web documents and embed them and search them, I have that system already set up using the GPU to go faster, so there’s no real disadvantage to having it also set up to spit out a small natural language summary using a shitty tiny local LLM to go along with the list. When running it on local files, you can totally ingest the local document stash slowly on even just a CPU and just store that and run the semantic search on CPU, no LLM required.

  • BioMan@awful.systemstoTechTakes@awful.systems•Stubsack: weekly thread for sneers not worth an entire post, week ending 1st November 2026English
    0·
    10 days ago

    Am I part of this category? I have a rather powerful home server that, among other things, I use to:

    • Run a decent coding LLM that helps me vomit up simple python functions for moving around lots of data between CSVs and graphing stuff, set up new docker containers with less trial and error, and run ML experiments on my GPU that previously took me hours to set up. I’m a biologist and previously I could run a commandline without breaking a computer but was very slow and non-native, now I can do simple non-security-critical things faster and actually understand novel error messages. Of note I never have that hooked up in a way that lets it run things just produce text that I can then do things with and maybe run myself.

    • Run my own local RAG system with embedding models and a small shitty LLM to search a hundred gigabytes of scientific papers and other PDFs semantically for things relevant to roughly paragraph length queries.

    • Run my own web metasearch engine with RAG and embedding models pointed at the internet, such that when I remember a paper I read fifteen years ago and provide a three sentence description it reliably finds it or finds lots of relevant old research (or old news articles I remember) when I give it a query and let it churn for five minutes.

    I feel like the important thing here is that I almost exclusively use them as information channels that take something at one end and spit it back out the other in a different or filtered form rather than making stuff, or effectively a piece of reference material for something that is formally verifiable and NOT CRITICAL FOR A SYSTEM TO FUNCTION. I also feel like all the actual innovations are more than a year old and do not understand people hanging excitedly on every little development.