Back in the day, the argument was that compilers produce unreadable assembly, so people used to writing assembly were arguing against the use of compilers.
Compilers also had bugs, so we still had to debug the assembly to understand how to fix the problem. Nowadays, almost nobody has to resort to those steps, except of course compiler developers. But that is just a testament to the quality of compilers.
Comparing LLMs to compilers is a take I often see, but I am not sure the comparison quite holds. The problem is that LLMs are inherently non-deterministic, so we always get a different output on the same prompt.
Maybe if LLMs are powerful enough it won't matter. I doubt it but we will see.
And in general, especially with something like C, the correspondence between the compiled code and source code isn’t very “lossy”. You don’t get one-to-one correspondence but it’s not too bad and you’re not introducing massive uncertainty by running it through a compiler. With LLMs though all bets are off… you’re not gonna know what you end up with
What you say is true, the comparison indeed doesn't hold.
But is it relevant? does it matter from a product perspective if LLMs are non-deterministic. You don't need to one shot the correct result, english is ambiguous and LLMs non-deterministic, but you can iterate.
If it's possible to iterate fast and cheap enough, even ambiguous language can produce the results you want, given enough iterations.
There are a lot of ifs and buts here, just a thought on the compiler argument.
I think it matters, because nowadays we don't look at assembly any more. I mean, I don't recall the last time I was tracking down a compiler bug but it's definitely been more than 15 years ago.
We do have to look at the LLMs' output, though, and, as you already pointed out, iterate to get the correct results. What this means is that the output must still be readable, must be analyzed by someone and I don't see it going away any time soon.
The problem is that the analysis is not cheap. Sometimes, with boilerplate, it is easy, but many times it is not and that's where we get only slight gains by using LLMs.
There is also a predictable relation between the input and output of a compiler w.r.t. the semantics of a programming language. Natural languages are ambiguous leaving room for the implementation to diverge that may not be obvious at first glance.
There are also plenty of things a compiler might not expose that end up being ambiguous. For example, C++ doesn't officially have a restrict keyword, so how can you express the fact that two memory regions never operlap and can be optimized? Unfortunately the reality is that if we had a language capable of truly expressing the programmer's intent we'd end up with a monster
Hand writing assembly produces more efficient and concise code, at the cost of developer time and required expertise. It was true for a long time, now not so much.
I think it's that when some code compiles to say assembler, the compiler doesn't prioritize readability and maintainability of the assembler code, since people are not expected to read and maintain it directly
Lots of really silly people love to compare LLMs to compilers. "You don't look at the compiled code either" and "Back in the day, people also had negative reactions to compilers and wanted to keep writing assembly by hand" and other such nonsense.
Indeed, the compiler does not have to ingest its own output, figure it out, and insert modifications in the middle. Source code is the medium that LLMs work in.
Human verification of the Lean program only requires verifying that the theorem itself is represented correctly. The theorem will only make up a very small part of the entire Lean program.
Wait, but we're doing that already, and it works well (Qwen 2.5 VL)? If need be, you can always resort to structured generation to enforce schema conformity?
In the demo, O1 implements an incorrect version of the "squirrel finder" game?
The instructions state that the squirrel icon should spawn after three seconds,
yet it spawns immediately in the first game (also noted by the guy doing the demo).
Yeah, now that you mention it I also see that. It was clearly meant to spawn after 3 seconds. Seems on successive attempts it also doesn't quite wait 3 seconds.
I'm kind of curious if they did a little bit of editing on that one. Almost seems like the time it takes for the squirrel to spawn is random.
If you don't have to pay for child care because you can just take off the time to pick up your kids from school, you are saving money your job otherwise forced you to spend.
Being forced to commute at peak time costs more - on a retail salary the difference between a peak and off-peak ticket can mean that the first hour or two of working is essentially pointless, as you're just paying back the cost of getting there in the first place.
Having to pay an extra surcharge to visit the dentist, because you can only go on a Saturday because you need to be at work other days. Flexible working would allow you to just take the Tuesday off no problem and go when it's cheaper.
I'm sure there are lots of other examples that apply to different lifestyles.
hours with your child aren't fungible. you can't pay the babysitter to go see the dance recital for you if you want to be the parent instead of the babysitter being the parent. all the money in the world isn't going to make up for missing the soccer game where your kid makes the winning goal.
Well to be pedantic, with all the money in the world you wouldn't be working for Ikea and the problem wouldn't exist so really that's a problem also solved by money...
Generally though higher paid employees tend to have more sway within a company structure and likely don't need to miss these important events, the win here is that something that was generally true for mid management up for most companies now extends down through all the ranks.
You can manage things in your life when they occur instead of spending money to displace them or risk losing your job because of them.
In another way, if you present a worker with the option between two jobs with the same hourly rate, one having flexible working hours and the other not, which would expect to be more likely choice? You can then measure the value of this choice by changing the hourly rates between the two until you see changes in outcome and you would be able to estimate exactly how much "more money" it appears to be "worth."
reply