> It currently generates 5–6 tok/s on an 8 GB M2 MacBook Air and 31–35 tok/s on an M5 MacBook Pro.
Where does this big a performance spread come from? I wouldn't naïvely expect SSD performance difference to be that big, and I would expect SSD performance to dominate...
The M5 SSD's performance uplift was fairly substantial, even when compared to the prior generation.
> In the Blackmagic Disk Speed Test, the SSD in the M5 MacBook Pro achieved read speeds of up to 6,323 MB/s, compared to just 2,031 MB/s on the M4 MacBook Pro. It's not like the M4 is "slow" in a vacuum, but the M5 SSD is over three times faster, which is a great generation uplift.
My suspicion is that this is simply due to the M5 having more memory, and the OS already having most of the file cached. The M2 has more memory pressure and would cache fewer of the SSD reads
If that's true, inference speed would be even lower if you have only 2GB total, including OS caches
I recently got an M5 Air as a second machine. I question if I'll even need a Pro machine in the future, as long as I can get enough RAM with the base chip (which seems unlikely, but...)
I am relying quite heavily on system caching and pread. And yeah, M5 is a way faster and I can guess Mac can cache something, even if process stays under 2gb.
It was 83ms read per token for M2 and 12ms on M5 pro. Total is 163ms/tok vs 30ms/tok for M5. So yeah, there is a faster read and faster gpu processing
Not only is it older, so Pro:Pro it would have much slower SSD, but doesn't the Air also have slower SSD than the Pro in the same generation? And maybe narrower memory bandwidth?
yeah, looks like a page cache matters a lot
I tested on mine m5 pro with 8gb memory pressure, got 27t/s instead of 35t/s
Someone tested on m4 max. In regular state it was 48tok/s, but 32-42 with memory pressure
Where does this big a performance spread come from? I wouldn't naïvely expect SSD performance difference to be that big, and I would expect SSD performance to dominate...