If you’re in the US, you might see a new shaded section at the top of your Google Search results with a summary answering your inquiry, along with links for more information. That section, generated by Google’s generative AI technology, used to appear only if you’ve opted into the Search Generative Experience(SGE) in the Search Labs platform. Now, according to Search Engine Land, Google has started adding the experience on a “subset of queries, on a small percentage of search traffic in the US.” And that is why you could be getting Google’s experimental AI-generated section even if you haven’t switched it on.
Yes training is the most expensive but it’s still an additional trillion or so floating point operations per generated token of output. That’s not nothing computationally.
Just consider how long it takes GPT4 to answer a question. Anywhere from a few seconds to a minute in my experience. There’s at least one A100 at probably 400w going full throttle that whole time, plus all the supporting hardware.