Fight this shit off. We do not need this. Digital Privacy has tools coming up - we are all loosing on this because of a few monkeys I wish some people had to courage to deal with. I'm honestly exhausted by this shit. Let all of it rot.
They can keep floating this stuff and we have to put in a thousand times as much energy to shoot it down. We get worn down quicker than they can. It only has to work once for them, we have to keep pushing back.
My very pessimistic take is one out of Taoism. If you wish for them to fail, let them grow. Eventually the weight of the whole system collapses itself, the question is just on what time frame.
The way out is to go through. I don't like this framing but it seems to be the most accurate.
You don’t even have to “let them” grow, they will do it regardless, as they hold the seats of power. But that doesn’t mean that silent acquiescence is the right path, either. Speaking up attracts like-minded allies.
It’s also a good idea to maintain your own private means of communication, as there will be some hard times ahead.
My heard hurts - i was stupid enough to think that SIMD was a CPU only thing - I don't understand why it would be ported to GPU - huge kudos to managing to surprise me
It's not really obvious unless you go in depth of the details on modern GPU architecture. GPUs aren't really SIMD, they're SIMT (single instruction multiple thread). The silicon looks a lot like SIMD, but the programming model is different.
If you go look at AMD's ISA docs (they're public) you'll see you don't have the equivalent of a __mm256 register like on x86. Each 'thread' just deals with single scalar values like int32 of float32. The hardware, however, groups 32 or 64 threads together which all run the same program and runs them together. Each 'thread' loosely maps to a SIMD lane. The SIMD is implicit, not explicit.
The main difference is that the 'SIMD' execution is somewhat opaque to the program. You just write plain scalar code and the hardware model dispatches it efficiently to SIMD execution units. It's not really an abstraction because to extract maximum performance you have to understand how it works. You can use this kind of programming model on a CPU too, Intel did it with [0] ISPC. It's a C-like language that has execution semantics similar to GPU shader languages but compiles to regular CPU code, and maps threads to your CPUs SIMD lanes like a GPU.
Yes, of course writing naive code assuming each lane in a thread group is a real thread is going to cause problems, but I didn't feel like I needed to go into that level of detail replying to someone just learning about GPU internals. I tried to cover this loosely by mentioning how you need to know how it works for maximum performance.
If you want to get more pedantic you also need to look at your target hardware and their specific micro-architectural quirks and features to get the best performance. AMD specifically benefits a lot from exploiting the scalar unit over the vector unit, you save loads of register file space if you can keep data in SGPRs over VGPRs. There's lots of traps you can fall into where you can load data from buffers into SGPRs but they get promoted to VGPRs because the scalar unit lacks an opcode for like one math operation you did to the value somewhere.
While each lane isn't truly a thread because it doesn't have its own PC the programming model definitely tries to make it seem that way. The threads can terminate at different points too. And again, the ISA isn't a vector ISA. Your register values are scalar.
> While each lane isn't truly a thread because it doesn't have its own PC the programming model definitely tries to make it seem that way. The threads can terminate at different points too. And again, the ISA isn't a vector ISA. Your register values are scalar.
This is not correct. If you check AMD's documentation there are explicit mentions of vector registers (VGPR), vector ALUs, and vector instructions. The introduction to Chapter 2 describes it as a vector ISA.
> RDNA4 shader programs (kernels) are programs executed by the shader processor. Conceptually, the shader program is executed independently on every work-item, but in reality the processor groups up to 32 or 64 work-items into a wave, that executes the shader program on all 32 or 64 work-items in one pass ("wave32" or
"wave64").
A VGPR is not the same thing as a vector register like in SSE4 or AVX. Each addressed register contains a single 32-bit value. A VGPR differs from an SGPR in that each thread in a thread group can have a different value in that register. An SGPR will have a uniform value shared with all threads in a group.
An add instruction on an AMD GPU adds two scalar values. If they're in a VGPR then each thread will add two values unique to that thread. A SIMD ISA as is common on a CPU is different because an add instruction explicitly adds a vector of values. xmm1 stores 128-bits of data. VGPR[1] stores 32-bits of data vectored over 32-64 threads in a thread group.
Without special instructions a thread can't access the VGPR values stored in other threads.
> False. If they were threads they'd have their own PC. They do not - only the warp has a PC.
They are using the term SIMT as it is normally used[1]. The "single instruction" part means that there is only one PC shared across multiple 'threads'.
Actually not so false anymore. (But still they don't expect you to use this knowledge while coding, and you should treat all threads in a warp as moving in lockstep)
> In GPUs of compute capability 7.0 and later, independent thread scheduling allows full concurrency between threads, regardless of warp. With independent thread scheduling, the GPU maintains execution state per thread, including a program counter and call stack... [1]
This might be more confusing than it needs to be. SIMD and SIMT are not mutually exclusive.
People commonly think of things like vector registers when they talk about SIMD, and each "thread" in a GPU warp definitely deals with local vector registers. Granted, they may be slices of superwide registers shared by the whole warp, or whatever else, but from the programmer's perspective, that's a valid way to think about it.
Put another way, it would be a mistake to think that each lane of a vec4 in a shader gets processed by a separate unit.
GPU "cores" are basically what a CPU would call SIMD lanes. So a GPU with 1024 'CUDA cores' might be structured as 16 relatively independent pieces that a CPU might call a core, each with a 64 wide SIMD unit.
When you dig through the CUDA developer docs instead of the promotional materials, you can develop a view of Nvidia GPUs as having 8-128 processing cores, each with 4 hyperthreads, running 32-lane SIMD for almost everything. Where a lane is 32 bits wide.
The promotional material likes to label the individual lanes as “cores” because it sounds more impressive. And, it’s not entirely incorrect.
Even the dev docs use the marketing terminology. The description I gave above needs a bit of piecing together.
This is a fantastic explanation, thanks for writing it. It also makes me wonder something: where exactly is the biggest difference between a 32-core x86 CPU (AVX512 basically being 16 32-bit lanes) and (say) an NVIDIA GPU with ~8-16 processing cores? Like why can't the CPU compete against a GPU like that for GPU-y tasks - or can it?
At risk of over-simplifying, GPUs are wider with limited computational expressiveness and higher memory bandwidth while CPUs are highly expressive computationally (and better connected to I/O) but with lower memory bandwidth. GPUs are less sensitive to memory latency by necessity. Even AVX-512 is highly flexible when inter-mixed with scalar code. GPUs get their very high register width by restricting what the cores are capable of doing efficiently.
Current CPU cores do two AVX-512 operations per cycle. If you can saturate this you’ll often run out of memory bandwidth on CPUs because of lower bandwidth compared to GPUs. In principle, if you bought a 192-core processor you’d have 6,144 GPU-ish cores of 32-bit operations, and they would run at a significantly higher clock rate than a GPU. It would not be competitive with a GPU for the kinds of things GPUs are good at it but it wouldn’t be as far off as you might assume. For some types of code, AVX-512 is unambiguously better.
Horses for courses. GPUs and CPUs were optimized for different things but their capabilities have slowly been converging over time. They all work from the same transistor budgets, the differences are where the tradeoffs are made.
There is a pithy silicon architecture tradeoff trilemma to be made regarding CPUs, GPUs, and barrel processors.
> Like why can't the CPU compete against a GPU like that for GPU-y tasks - or can it?
Others have taken a stab at the actual differences, but there is a deeper fundamental reason.
A CPU is optimized for low latency of operations. They are designed to complete a given piece of code as fast as possible. There are some affordances for throughput, such as SIMD, but even those are designed to only be as good as they can without compromising the low-latency design of the core.
And the reason this cannot compete with GPUs in throughput loads is that after a point, completing a single task 2x as fast costs a lot more than 2x the transistors and power. CPUs chase that curve as high as practical, GPUs stop once it no longer makes sense for throughput. This is not just clock speed (though it is also clock speed, modern GPUs hang around in the 2.5GHz area while CPUs are about twice that), but especially their ability to hide memory latency, and ILP. CPUs spend big on being able to issue, execute and retire multiple instructions from the same stream, with complex reordering and more than half a dozen execution units per thread, while GPUs are either scalar within a thread, or maybe dual issue. A CPU has a cache hierarchy optimized for bringing average memory latency down, while GPUs just juggle more threads and use them to get something to execute when waiting for memory.
The major one is that there's one layer of indirection that exists on GPUs that doesn't really on CPUs. There's one giant vector register file per for each of these processing cores (that'll be something like 2048 rows x 32 lanes x 32 bits). An individual shader invocation might only need say, 16 rows. While there's hardware for issuing 4 hyperthreads at any given time, there can be a variable number of thread states in the register file. So for the case of each invocation only needing 16 rows, you might be able to fit 128 hyperthread states into processing core. Those four hyperthreads then hardware schedule those 128 states and will execute any that are ready, as well as allocate more from other scheduling hardware as gaps in the register file appear from shaders completing.
Because of this massive amount thread state, you don't depend nearly as much on a cache hierarchy to deal with DRAM latency. There's ostensibly some other thread state sitting around that can be serviced while others wait for the hundreds of cycles of latency to access DRAM.
So the whole model of how you account for the discrepancy between ALU cycle time, and DRAM latency changes versus a CPU. Where a modern CPU spends a lot of area on complex cache hierarchies, speculation, etc, to hide the latency to memory, a GPU focuses on having a lot of thread state around and a lot of ALUs, but balanced ideally, so there's always ALU work to do while other thread states are waiting on memory.
Now, over time, GPUs have gotten more complex hardware, and more complex cache hierarchies to cover the cases that aren't handled well by extremely long access times. But those tend to be very explicit. Additionally, CPU vector files have gotten more similar to GPU cores as architectural features like lane masking/predicates have been added to have the equivalent of CUDA threads in the same warp that take different paths through control flow blocks. That's a lot of what people mean when they say that AVX-512 adds a lot more than just 512-bit registers. The K mask registers let you do a lot of GPU shader tricks to have effective partial residency, and not have to use all the lanes if the data doesn't line up with that.
If by processing cores you mean Streaming Multiprocessors (SMs), keep in mind that each SM is 128 threads, so 16 SMs is 2k threads - or 2048 single precision math ops per cycle. The 32 core avx512-enabled CPU is 512 math ops per cycle, if you have one FMA unit per core (or 1k ops/cycle if you have 2 FMA units/core).
Note that modern NVIDA GPUs like the 5090 are actually more like 170 SMs on a chip, or 21,760 flop/cycle, or ~20-40x more ops/cycle than your example CPU.
Put GDDR6 vs DDR5 memory on top of that, and it’s easy to see why the GPU can churn through math so fast … as long as it’s GPU-y. For stuff the GPU does well, the CPU typically can’t compete, the GPU is often more than 10x faster. But GPU-y tasks are a subset, and there are CPU-y things the GPU can’t compete on, despite (or even because of) the thread count discrepancy.
On a CPU, hyperthreads are mostly replicated register banks. This allows the CPU to hold the context for 2 threads simultaneously. And, lets parts of a CPU make progress on one thread while the other thread is stalled. CPUs also has a kinda large microcode register bank that helps work around dependencies in asm instructions that reuse named registers.
On the GPU however, the hyperthreads are just a round-robin execution queue to take advantage of instruction pipelining. The register bank of a single GPU core is huge and can be flexibly divided across a variable number of thread contexts when a kernel is launched. Many thread contexts can be held in registers simultaneously in a single GPU core. That makes stalling on memory latency much less of a problem. The hardware can focus on delivering raw bandwidth with high latency and get great overall performance. This throughput-instead-of-latency trade-off extends to many other aspects of GPU design.
SPIR-V states its an int, float, vector n (where n <= 4) or a matrix (2..4 cols of vector n).
It does not necessarily mean the hardware can do 4x4x64 floating point operations in a single subgroup operation, but at least the programming model supports framing it that way.
There is something very SIMD-coded in GPU programming which is coalesced stores/loads, if a warp (32 threads) handles contiguous memory, it will create ~4 transactions instead of 32.
Welcome to the lucky 10,000! SIMD is actually a pretty integral part of how GPUs are able to work efficiently, it's part of why there's such a strong focus on branchless programming in the field.
I built myself such a simple lexicon for technical stuff (concurrency vocab - invariant, genetics, stuff like that) - can only recom the practice as vocabulary is clearly a big step difference
2 days ago in the night, French President Macron has passed through the mandatory age verification on social networks to force identity verification.
Anyone with good sense of pragmatism would come up with the reasoning that forbidding a smartphone to <15YO is a bounded law with more effect on the targeted demographic.
This style is of course a lie is disguise to push for taking control on the networks and news outlet. I can imagine how news in the US are (fake) - and you can imagine that we as well have news outlet completely fake in France.
The internet has given us a hope, it has given results to democracies, we are seeing the world for what it is, we have much more people that are aware of the broken system, much more people that are transgressing from the mainstream outlets.
We were given the right to inform ourself by ourself, to get to criticize things. And this is slowly being taken out from us in Europe.
I'm devastated atm. Quiet honestly very emotive.
We need to talk about this. We need to stand up. We need to do it, right now. I cannot think about a world where someone would decide what i read or what i'm going to think, or what i can say or not, and who to talk to and why - i do not understand why people would be guilty by default.
I posted this freaking topic on multiple french subreddits to get this kicked-out on all of them - it's arguably one of the most important topic of the hole f* internet.
Benchmarking like never before
reply