Wasn’t Huang the guy who said your $500K employee should burn at least half his salary on tokens? I don’t think Nvidia is focused on cost-effective general computing for anybody.
It is possible to serve different markets with different agendas and have both of them serve nVidia’s business. Think of all the freemium software out there; widespread so everybody can use it, but the people who REALLY use it REALLY pay up.
Maybe but they make their real money in data centers. Certainly, they want to have fingers on every trend. Owning something the Government wants to control gives you more access to trade repression of open models for more Government favors and money.
Nvidia is halving the production of its higher end GPUs, doubling prices and heavily segmenting the market. They don't give a single shit about individuals. One wafer that makes 10 5090s that might sell at 3000 each, or one wafer that it already sold 6 months ago for 200k to one of the incestuous AI companies it works with?
There already are a lot of forks for things they decline to implement. TurboQuant, ROCmFPX, and more. I need to set up an agent that will loop on merging them.
> "They’re not capable of understanding a brilliant language but we want to use them to build good software. So, the language that we give them has to be easy for them to understand and easy to adopt."
This makes it a great language not just for young Googlers programmers, but also for LLM Agents!
IMO, the next big language will be similar philosophy, but without garbage collection. (is Zig the closest to filling that niche?)
- Strict subset of go might be confusing to an agent actually (trying to use unavailable go features)
- So -> c11 source to source compile loop might be confusing to agent: if So compiles to c11 does it guarantee c11 program compiles. If runtime exception (segfault etc), is it going to be easy for agent to map that back to original So code?
In the context of local LLMs on limited hardware I've ran to the exact same conclusion: "tok/s" isn't the most useful metric when my personal North star metric, given my fixed hardware is: Model smart enough to execute my goals _in the minimum amount of time_.
Some models I tried (Mistral I think) had better tok/s, and roughly same billion parameters / scores on various benchmark... But they were _so_ verbose, that they generated many more tokens compared to a Qwen model of same caliber to answer the same thing.
So even though it had better generated tok/s, because so many more were generated, the clock time was longer.
And this compounds over mutli-turns: more generated token means more context used in the next turn (until some compaction or something runs)
Even more important in a local context is the difference between token generation and prompt processing speed. We tend to focus on the former, but for multi-turn/agentic workflows the latter can dominate.
I feel like we need to see more proliferation of local LLMs to start seeing ones turned to be terse, rather than maxing the amount of tokens user pays for
The huge spike of "lk-99" in science & frontier tech is amusing...
This is cool concept, would love a positive/negative sentiment computed for each comment that refers to a given word, so you can see trends of "cloudflare (positive)" vs "cloudflare (negative)" where first one counts comments only if sentiment confidence is greater than say 0.6 and the other one counts comments only if sentiment is less than 0.4 (assuming [0,1] sentiment score)
> About the generation speed: ~100-150 t/s on the RTX 5090 and ~40 t/s on the Mac
Curious if you can share the prefill speed too?
I run locally on a crappy desktop (some AMD iGPU with Vulkan llama.cpp, 32 GB DDR4 RAM) for experimentation. I get 15 tok/s on generation for the qwen & gemma4 MoE models. I get around 150 tok/s prefill speed.
Reason I'm asking about the prefill is looking at my stats at work, I use between 20M to peaks of 300M input tokens daily. Some of those token are cached but in general, I seem to have roughly 500x more input tokens than output. So interested in prefill tok/s stats.
Btw, based on your numbers, I think our use cases are quite different. I use the agent for very targeted sessions - basically things that are clear to me how to do, just want to automate them. My workflow is usually: new session -> read this, this and this -> do that. I.e. I don't let it wander at all in the codebase, so I rarely exceed the context window.
Also, I get a lot of mileage from the ngram-based speculative decoding functionality [0] as it allows me to iterate on the implementation much faster.
I do use it the same way as you're describing on personal projects at home, in a very crude manner (pasting code snippets in llama server web UI prompt. Next will attempt OpenCode)
At work I use it in similar manner with more mature tools, but the vast majority of token spend comes from a totally different workflow: "pretend the AI is a fleet of junior/intern engineer you're delegating work to", where the agent will on its own do the implementation, commit the changes etc.
It does indeed spend a lot of tokens wandering the codebase, talking to MCPs, loading skills etc.
> Quantization-Aware Training (QAT) [...] allows preserving similar quality to bfloat16 while dramatically reducing the memory requirements to load the model
In ~2015 got an Xbox one, as a media center it was an awesome experience:
Kinect voice control to play/pause and other things way before Google home/Amazon echo ecosystem were mature.
Free OTA channels via TV tuner and well designed OneGuide (with ability to pause and rewind).
And of course all the Netflix and other apps, Plex server etc.
But strategically it seems Microsoft decided they wanted to look more like Playstation, focused on gaming (at that time paid Xbox live subscription vs free Playstation)
And as gaws points out, they seem to recently announce to double down on the gaming stuff.
So when they discontinued OneGuide. I picked Roku since they seem to be focused on the media experience primarily... but unsure how I feel about this acquisition news.
Since parent mentions "toxic byproduct": Say you're the company that invented Teflon pans. you made billions. You saved billions in time for all the users of the pans... A true entrepreneurial success.
But, by how many billions did you fuck up the environment, people's health etc with the spread of PFAS everywhere?
(Ggml.ai is llama.cpp.)
Curious if the “I consider HuggingFace more "Open AI" than OpenAI” sentiment in that top comment will still apply with NVIDIA as the boss now...