I think the most useful word in both cases is "extrapolating".
An LLM extrapolates from its context window to the immediate next token. This word applies whether you view what's happening as "reasoning", "prediction", or as a math function.
If you're assigning steering 70 of your 100 output points because it's what you think we should go with most of the time in this situation, I'm going to call that a prediction of how to steer.
The point was, if your internal model of the world makes a prediction of a negative outcome at some point in the future, and you optimise your individual actions to avoid that negative outcome, then wouldn’t it make sense to focus on the fact you’re building and optimizing towards an internal world model rather than the fact you’re executing your actions one at a time in series?
Yes, but I think the same construction could also be used to characterize the first system; it determines the next move based on a prediction of its reward signal, where its reward signal is a measure of how likely it is that a grand master would make that move.
Like stanleykm, I found this analogy somewhat puzzling. On reflection, I think the author's point is this: the statistics of actual usage do not seem sufficient to produce a fluent LLM; it also takes reinforcement learning.
A classically pretrained LLM does not have a concept of having determined its previous tokens — it has only ever observed inputs that it had no causal influence over. This is why it's valid to say its actions are predictive and not determinative.
If you grade what was linked by whether it is "approximately achievable today", and not some less interesting metric like being predicted for the correct year or if it was outcompeted by some other thing, he's closer to 70-90% depending on how close you're willing to grade.
It's very easy to say 'person X made a highly specific testable prediction, while respectable people said nothing like it would ever happen, and it only 90% happened, so person X was a fool unlike all the respectable people', but it's a trap. In reality Kurzweil was directionally correct about most things, overspecified the details, and had optimistic timelines in the way that everyone has optimistic timelines about everything.
> while respectable people said nothing like it would ever happen
Can you give some examples of predictions he got roughly right where this applies?
(Not a gotcha, I'm genuinely interested, because this is the key for me when thinking about whether to give credit for 'close' or 'right but early' predictions. If you're predicting things that most others have dismissed as impossible, or nobody has even thought of, then it's pretty impressive and interesting when you turn out be even roughly correct. (I'll still ding your credibility if you are overconfident about dates and details, but I'll do that while paying plenty of attention to what you say next.) If you're predicting things that are already suspected to be possible, though, and what makes you stand out is your confidence and your timelines, then I'm not going to be very interested when some of your predictions turn out to be fairly close to the truth.)
2020-2050: Phone calls entail three-dimensional holographic images of both people.
This is totally possible. We could even do it on phones with fairly mundane consumer technology. We can do it with glasses, even. People just don't care. The prediction has yet to land but in spirit is correct.
Centuries hence: Computer intelligence becomes superior to human intelligence in all areas.
Anyone doubting that this will be true within centuries is nuts.
2009: People can talk to their computer to give commands.
At most a couple years early in technicality, and in spirit over a decade early.
2009: Computer displays built into eyeglasses for augmented reality are used.
True today, if not a particularly popular product, and later than suggested.
2009: A $1,000 computer can perform a trillion calculations per second.
Definitely true today. I think this was basically on time, too.
2019: Most people own more than one PC, though "computer" no longer means laptop or box-plus-monitor.
Freebie.
2019: Most learning is via adaptive courseware presented by computer-simulated teachers; human adults are counselors and mentors, not instructors.
We obviously could do this today, though it might not be a great idea for the students. Early, and socially blind, but basically right about possibility.
2019: Prototype personal flying vehicles using microflaps exist, primarily computer-controlled.
Basically wrong. There are eVTOL companies aiming for this, and people do have camera drones, but the sense it was meant wasn't predictive.
2019: Human-robot relationships begin as simulated personalities become more convincing.
Early, subscale, and fought against by providers, but this is a thing.
2029: Massively parallel neural nets constructed by reverse-engineering the human brain are in common use.
Ok people will mob me for saying this, but this was more right than wrong. Definitely at least a bit wrong.
Roughly, these satellites would be as bright as the sun, except at night when you're adapted to dark conditions. With the naked eye it'd just be unpleasant, because the dot is so small and smears out to a lower effective brightness, but through a telescope it's bad enough you could get eye damage fairly quickly.
With Starlink, I thought the more extreme complaints were unreasonable — astronomy is important, but it's not so important that marginal costs are unpayable — but with Reflect Orbital we're talking about costs like 'it's sometimes uncomfortable to look at the night sky' where it's really hard to imagine a counterbalancing win.
Indeed. Humans and not other apes reached the moon by our great proclivity to long distance running, through our exceptional sweat glands... ah, wait.
How is it possible to live in the world around us and not see, apparent and transparent, that intelligence is the lever on which everything rests? Humans are surprisingly weird animals, but it's not our hairlessness or our fairly average propensity to violence that did this. We don't have more houses than chimpanzees because we're _strong_. We haven't defeated the tolls of disease because we're atypically devoid of restraint. We don't have more effective governance than termites because we're more communal.
Of course intelligence is the main bottleneck. The arguments here are so confused. What do you think regulation is made of? What do you think money buys? What do you think determines how well you can gather data, or how efficiently you can consume it? But of course, as the genre, after spending the first third waxing about how its opposition could be so foolish, the last third is spent finding a pretty insult and wallowing in it, so as to not to leave enough space for the brief argument in the middle to consider such things as why someone might disagree.
It's very clear that we have plenty of intelligence but we are not acting in it. What intelligence are we missing that is needed to solve climate change? (our most pressing problem - which AI is exacerbating)
> We don't have more houses than chimpanzees because we're _strong_.
You're literally making the point that we already have intelligence. I'm not sure how this is supposed to be an argument for a shortage. To revise: The point you are arguing against is not that intelligence is not important but that it's not the bottleneck.
> What intelligence are we missing that is needed to solve climate change?
Cheaper solar and batteries? Effective drilling for geothermal? Working cost-effective fusion plants? Earlier awareness of the issue? Better coordination and communication? Better dealmaking? Better support of laws for voting systems that reflect broad preferences and that internalize externalities?
What answer could you possibly hope there be, other than intellectual ones?
> You're literally making the point that we already have intelligence.
?? So do chimpanzees? The existence of intellectual capability does not preclude returns from greater intellectual capability.
The perfectly rational actor myth in economics will have Mary doing all sorts of things she doesn’t actually do. Mary just really likes peaches and the whole construct of rational actors is a fictional superstructure. Same with this idea of intelligence levering all our behavior.
Maybe Mary keeps intelligently finding new ways to find scarce peaches, but her ultimate benefit is eating the peach. It’s that human desire if anything.
My claim that Mary's extraordinary ability to get peaches whenever she wants is due to humanity's collective intelligence is not at all a claim that Mary wants peaches because she's a pure rational actor. (??)
How much more intelligence will it take to get people to be ok with data center construction? How much more intelligence will be needed to reduce overuse of antibiotics? How much more intelligence is necessary to enable free trade with Canada?
What property, if not intelligence, do you think makes people capable of rallying political support? Of working quickly though permit processes? Of providing accurate drug dosing guidance? Of presenting as a trustworthy personal healthcare guide? Of negotiating political deals? Of making effective public communications? Of juggling independent parties' interests? Of coming out ahead in contracts? Of producing alternative antibiotics? Of detecting and handling drug resistance in the wild? What answer could your questions possibly have other than this?
As far as I know humans with the same sort of brains as your brain and my brain have existed for 100,000's of years, perhaps as long as 2 million years.
They did not go to the moon. They did not make mechanisms, they did not make pottery, they did not smelt iron, they did not press shapes onto clay and leave it in houses that burned down, they did not have houses.
All of that has come in the last 12,000 years. I think that there are people in the Amazon today who could not write or read a word, could not drive a car or solve an equation or argue a case or compose an algorithm who are as intelligent as me, or you, or more. I think that people like that have existed for more than a million years.
So if they were so smart, why didn't they go to the moon?
In what way do you mean iterations - what's your model for iteration as a mechanism for iterating intelligence? Do you mean that intelligence improves with every generation?
I mean iterating to build the technology required to visit the moon, starting way back with fire and a stick. No different than an LLM iterating to build software. As you said human fluid intelligence has likely not changed significantly.
Seems intelligence is actually a bit antithetical to the idea of a stable society. Look at the rate of change of our species. Within our own lifetimes there is so much change that it eventually exceeds our abilities to property mental model this change by the time we are elderly.
Meanwhile, consider the termite. Their society has been stable for millions of years and they've colonized the earth. Intelligence is not needed for that.
I love when tech bros think they can make up a new groundbreaking theory about a topic, in this case economics, they demonstrate zero knowledge in. You're definitely projecting on the confusion point
do you remember that article from google where it would hallucinate as if it was on acid/psychedelics and people going crazy saying this is the AI "imagining reality" and then shortly after some Googler went on a very brief media tour saying LLMs were sentient?
its crazy how much people extrapolate, the consensus back then was "cute but we won't get there" then suddenly we got GPT 3.0, 4o, 5.x, seedance
what people really underestimate is how much faster progress is now with AI
The simplest explanation is surely that China is new to having a highly educated workforce and the ‘failed for some fifty years’ claims the article makes don't mean much given it.
I don't think you need to entrain market arguments or whatnot to this, when it's only in the last decade China was a strong advanced manufacture player, and turbines are the kind of project that you probably wouldn't expect to go much faster than that regardless of the demand.
Exactly. Advanced turbine manufacturing is largely a chemistry and metallurgy problem. The Great Leap Forward and Cultural Revolution killed off China's intellectual capital so they had to start from scratch in 1976, and only really got moving in 1992. They've been way behind but will catch up eventually.
According to the article, the skills gained from decades of manufacturing experience and the yield capabilities are more critical than simple chemistry and metallurgy.
Quick question. Same issue in Japan. They have talents and capital -- why do you think they are not able to come up with a reliable turbine -- when they can manufacture almost everything else?
Economics. Japan has chosen to specialize in other areas and being a jet turbine engine manufacturer isn't where they've chosen to put resources in. The pay off simply isn't there. They're there in the supply chain, with IHI, MItsubishi, and Kawasaki being in the business of advanced titanium and nickel superalloys being used in wind, steam, and gas turbines for powerplants. It's just that the economics of commercial jet engines themselves isn't something they've chosen to pursue.
Because repression is a thing of the past in China? They continue to persecute investors and business executives who get too successful and to repress academic work they find politically objectionable. The communist party continues to exercise control over all intellectual endeavors.
And don't think it doesn't matter for jet engines. This article gets into how the open acknowledgment and dissection of engine failure in the West promotes quality and that China has clearly not adopted this culture. Of course not.
They'll catch up in the same way the Soviet Union used to - at unsustainable cost and on the way to falling behind yet again.
Persecuting or favoring investors on a political basis "and to repress academic work they find politically objectionable" has come to America. The only business people intellectually weak enough to not be a threat are real estate developers.
Not that the tech industry leaders who are way too comfortable with fascism are making such a good showing either.
So you think it’s just as bad as China here? Please state plainly instead of dodging around. Would you just as soon live there, have made your career there? You think you’d be able to publish your books and prosper there? You think we are fascist and so… your life will be miserable now?
I think you’re being absurd. You’ve had a great and largely unfettered career. How has repression hurt you in the slightest? Who repressed you?
Plainly, you’d much rather live here than there, and your equivalence is totally false. It’s a dictatorial repressive regime and much worse than here and you know it, and your comment is a corrosive disingenuous false equivalence.
It’s really a bizarre statement, a sort of narcissistic need to feel persecuted amidst a life of plenty in the wealthiest society ever known. They have forced sterilization of Muslim women in China, up until recently forced abortions under the one child rule, and you get disappeared for criticizing dear leader. And your cozy life is comparably miserable? Only in your mind.
This style of argument has always bothered me, because the correction to misdiagnosis or mistreatment is not to stop looking, it's _git gud_.
For sure, we have to be realistic about what processes will systematically have error, and if we can't stop a doctor from doing bad things with a piece of data we should shield them from it, but the tools to make scalable, calibrated risk estimates based on large data dumps is getting better every year.
There are physical limits to detection and technical parameters that make some situations indeterminate even for the best of the 'gud'. It is frustrating that, hearing an argument from many different individuals over a long time, you assume that each speaker is missing the critical insight that you possess.
> but the tools to make scalable, calibrated risk estimates based on large data dumps is getting better every year.
So your suggestion for indeterminate scans is more scans? There is no 'large data dump' personalized to you except for your own imaging.
> if we can't stop a doctor from doing bad things with a piece of data we should shield them from it
The doctor isn't the problem, it's the people who would be seeking out monthly imaging without symptoms
I go to the doctor every year for a checkup without symptoms. Why a year? Why not every six months? Two weeks? Day?
If the false positive rate is demonstrably low, I can't see the risk. People who think they need a doctor will go to a doctor with or without a fancy scan. People who want to play armchair physician will play armchair physician with or without a fancy scan.
> If the false positive rate is demonstrably low, I can't see the risk
The false positive rate is the entire risk.
When you go to the doctor for a physical they don't run all of the blood tests they can. They only run them for specific symptoms and for specific preventative measures where we've calculated that the benefits outweigh the risks of a false positive.
Some tests have been removed from routine exams, or at least discouraged, because they were producing more false positives and harm than what they were saving.
Full body scans are deep on the end of the spectrum of tests with high false positive rate when ordered without supporting symptoms. That's the risk.
> People who think they need a doctor will go to a doctor with or without a fancy scan. People who want to play armchair physician will play armchair physician with or without a fancy scan.
Not really how it works in real life. When you get a full body scan, especially with ultrasound, there are a lot of benign things that can show up that vaguely look like non-benign things. Even if the interpretation is "probably nothing", many people start worrying and think they need to get more tests just to be safe. Even people who don't see themselves as "armchair physician" will start thinking that they should at least rule out the worst case because they wouldn't want to die of cancer having known that something might have been there.
They only run them for specific symptoms and for specific preventative measures where we've calculated that the benefits outweigh the risks of a false positive.
True to some extent, but you're ignoring the role that costs and insurance play here. Do you really think the personal physicians of billionaires and heads of state are only running a limited set of blood work because they're worried about false positives?
You can get scans without your normal doctor recommending them. The point is that there is evidence that scans obtained ‘just because’ are harmful as they lead to unnecessary procedures at the population level
More often it leads to people thinking they have issues when they don't.
The same thing happens with blood tests: You can order all the blood tests you want if you're willing to pay for them. If you order enough, you will get some that show up as abnormal. You can start spending tens of thousands of dollars ruling things out and never catch any real issues.
I’d want to see the data, and even if you had 10x the rate of false positives (to true positives) that resulted in unnecessary tests and procedures, it still could be worth it, depending on the severity of what you avoided with the testing.
I actually don't think we have the data available that I want, and even if we do, as many others here have pointed out, intentionally sticking our heads in the sand forever makes no sense.
How do you get the false positive rate low? There's a lot of things that look weird on a scan that turn out to be benign. And if you tell patients "well the chance this turns into a serious disease or cancer is low but you can get this optional procedure to fix it now if you want" how many do you think will take them up on it?
A new chargeable procedure is for for the hospital but maybe not for patients imo.
Many countries with far better outcomes don’t do this, is it necessary, or is it just the product of an insurance-driven health industry which prioritises interventions over health?
Maybe there is a bias for action within our moral and legal system. Fundamentally if you can deal with uncertainty correctly or "perfectly" wouldn't more information always be better?
I have libertarian enough tendencies to think that if a person wants to self-operate, or pay for an operation that doctors are telling them is not justified given the evidence, then they should have right to do it. But I don't think that's what people normally mean when they say that eager screening causes harmful overdiagnosis.
> So your suggestion for indeterminate scans is more scans?
The solution to imperfect evidence is consistent and calibrated risk estimation of both disease and intervention.
The risk estimation is why people aren’t recommended to get scans! There are studies on ‘VIPs’ who get ‘executive MRIs’ and wind up getting treated for things that would never have justified intervention.
Isn't the way we decide what justifies intervention by comparing observational data, action and outcomes? Currently our observations are limited by many things including the cost and side effects. More frequent or better observations will improve the assessment of what justifies interventions.
That sounds more like a capitalism issue, to be honest. Treatment = revenue, so of course there will be unscrupulous individuals who will bend their oath and let patient anxiety drive care.
The trick seems like it would be to strongly incentivize waiting and watching any symptomless anomalies if further investigation is invasive. If you're getting 60 second scans every month then something growing will be catchable and something static or that disappears can be ignored until the next scan.
exactly correct. if a bit of knowledge is dangerous, the correct response is not to choose ignorance, it is to get more knowledge about what dangers arise and problemsolve some more there. run it out a few hundred years and it is then no longer dangerous, and strictly better than ignorance.
If Midjourney says "maybe you have cancer" but your doctor doesn't take it seriously, you might sue if you do end up with cancer. You might even win, regardless of whether "wait and see" was the right approach.
Meanwhile, if your doctor gives you an unnecessary CT scan that rules out cancer, hospital both earns $$$ and the doctor doesn't face legal consequences. Your increased chance of cancer risk from the radiation isn't something you can realistically sue over.
No one is saying that we should stop looking. Especially not the commenter you replied to. They're saying the tech Midjourney presented isn't _gud_ enough to justify frequent scanning.
Consider the null space of diagnostic markers, say, the precise shape of a tissue boundary used in early cancer diagnoses that comes out blurry in an imaging system every time. More scans with the same null space will not resolve the null space.
> This style of argument has always bothered me, because the correction to misdiagnosis or mistreatment is not to stop looking, it's _git gud_.
Exactly this. I mean, even if the scan is really indeterminate, at a minimum you can simply wait, then scan again. If it's truly something serious, it will become determinate at some point. Doing this is still better than nothing and carries no risks of unnecessary procedures.
Wow this paper is bad. I was expecting little and received genuine crackpottery.
It's hard to critique this paper directly because its claims are so incoherent and decorated with so much obnoxious verbiage[1] that people aren't going to believe me when I point out what the claims actually are.
Regardless, this is their paper:
First, they conflate the substrate with the presentation layer. Then, they point out that Turing equivalence means you can run an LLM on anything, with a pointless aside where they nerd out about making a logic gate in AoE II. This lets them conclude that you can use anything as the presentation layer.
Then they claim that it's natural to ascribe human-like attributes to outputs from some presentation layers, like abstract letter symbols on a computer screen, but not to most other things, like patterns of goats on AoE II, or LEGO. Yes, this seems to imply if your partner writes something heartwarming to you using LEGO, you're meant to laugh at them and point out how LEGO isn't intelligent so this isn't evidence of anything.
Then they do a thing where they say that assuming substrate independence is true (or false) prevents proving whether substrate independence is true or false, and from this, but just by vibes AFAICT, make it sound like everything one could learn about attributes of a system from its outputs is circular.
Then they write a bunch more incoherent text and mercifully then it ends.
[1] 'from an epistemic perspective, we argue
that a generalised conclusion such as that necessarily requires a well-designed experiment' — the whole thing is like this.
They don't even implement their logic gates within the normal game mechanics but with scripting some bit-goats in the editor. So the AoE2 Engine is just a graphical representation of their script.
But my favorite is this one:
"Corollary 1 (AoE II is Turing-Complete). Let I be an instance of AoE II with two players p0, p1. Assume p0 has two markets, a town centre, a trade cart, six villagers, and five farms; while p1 has a scout unit and only attacks p0’s buildings. Then if I has no time or size limits and the terrain allows for buildings everywhere, the game session in I is Turing-complete."
Why being so explicit about the setup with no further explanation? Isn't it anymore turing complete with seven villagers and six farms? Is it even possible that a player can trade with himself?
reply