Would it work as well, though? The basic idea of "degrowth", to me, is to slow down the car until the braking distance is smaller than the viewing distance, so we're safe no matter what. Whereas technology is more like swerving constantly to avoid obstacles. The latter is just intrinsically more likely to get us into a crash.
I'm not saying degrowth is feasible, mind you. I agree that it probably isn't. But the fact we can't do something doesn't mean the alternatives must be good. It might just mean we're cooked.
Technology did that. With lots of government subsidies, and in particular with Germany kick starting a ton of solar industrial development.
Even in the US, with concerted anti-renewable sentiment at all levels of decision makers in both electric utility leadership and government leadership, new electricity is overwhelmingly carbon-free, because the technologists made it the cheapest:
Technologists caused massive social change, by changing the technology. Far more than the people that focused only on culture and changing society through those means.
Sure. That's good, I fully agree with you on that. But is this the solution, or just treating the symptom rather than the root cause?
Technology by itself has never been the problem, the problem is the scaling up of it. Fossil fuels are harmless at a small scale, it's only once you exploit enough of them that you start to see harmful effects. But renewables are no different: we're drawing from the Earth's natural energy allocation, but that's the same energy that heats the planet in the first place -- so as we scale up renewables, we will also end up wrecking the planet. We'd need to scale it up so much it feels preposterous, but it seems obvious to me that if we can, we will. There is no upper limit to energy demand.
And that's what I'm worried about in the long term, I suppose: that surprised Pikachu moment where human ingenuity (and AI, I guess) worked so hard on extracting every bit of solar energy that the rest of the planet, not receiving any of it anymore, predictably turned into an ice ball. This kind of nonsense could never happen if we could properly look and plan ahead as a society, but neither would climate change, so.
Don't get me wrong, it's great that renewables are booming, but we could also have built a sustainable utopia on the back of fossil fuels if we had properly budgeted their use. We need to switch ASAP and start budgeting, because the problems we currently have aren't the only problems we will ever have, and the size of them, unfortunately, tends to scale with energy.
This. People treat "token prediction" dismissively, as if it were a limit and not a foundational skill. Human brains do "token prediction" in all sorts of contexts.
I think that if today's capabilities were explained to someone 10-20 years ago they would think this is definitely AGI, but they would also have expected much more disruptive changes to society as a result than what is happening. I figure that's because we have abstract intelligence without physical/grounded intelligence, and it turns out the former isn't general enough to implement the latter (remains to be seen if the word after that is "yet" or "ever"). So I think we do have AGI as conventionally understood, but our understanding needs recalibration.
> but they would also have expected much more disruptive changes to society as a result than what is happening.
> I figure that's because we have abstract intelligence without physical/grounded intelligence,
I put the cause on "not enough time". As a thought experiment, if an AI today were to (miraculously) produce a cell design template for a cell that, when injected into somebody's brains cures their Alzheimer's, how long would it take for that to reach the clinics? The actual physical tech barely exists, and let's not forget about the regulatory quagmire. So, with some optimism, I give it about four decades. In the same four decades, the same AI in the hand of unscrupulous actors could bring enough devastation so many times over that we may need to enforce a global ban on AI. In any case, I'm pretty sure we are going to get our disruptions; it's just a matter of time.
The problem with that perspective is that people thought, "Only AGI can do X, therefore, if a thing can do X, it's AGI." Because they can't imagine how X could be accomplished without it.
However, what's actually changed is how people perceived X because we don't have to imagine. We understand now that it doesn't require AGI so we no longer make that leap to assume it's AGI if it can do X.
It's really going to be a "I know it when I see it" situation.
We've underestimated how long it is going to take to validate and build into some of the most valuable areas, and probably overestimated how much new CRUD software is needed (or people are willing to pay for) I think there is still a lot of room in the tail for custom software, but the niches are tight!
> globalization should increase linguistic complexity via exposure to other cultures. I certainly didn't use the term chaebol until the rise of Korean pop culture.
Globalization moves every culture towards each other, with a higher weight on dominant cultures. You borrow a few Korean words, they borrow a few more English words. McDonalds pop up everywhere, everyone buys the same phones, hotels across the world converge towards a standard experience, and so on. The end result of such a process is a monoculture. Not now, not medium term, but I expect a few centuries of globalism would suffice to blend all extant cultures into a slurry.
I feel that the main issue with the rationalist crowd is that they live too much in the space of rationality, intelligence and abstractions, but not enough in reality.
This leads to an outlook where everything must, almost axiomatically, be intelligible; reality is subordinated to intelligence; and no matter what is real, intelligence can prevail upon it and bend it to its will. Whereas I would argue reality is actually pretty resilient to intelligence -- sufficiently chaotic systems are largely unintelligible, the distribution of energy and resources is fixed and can't be magicked into being, and intelligence appears to be most effective when there is a clear observable feedback loop to keep it on track, which is an external bottleneck.
So it's not necessarily specific predictions that are off, but the implied consequences of these capabilities. Yes, these systems are uber smart, but uber smart people are rarely particularly powerful, so... does it matter? It depends on how powerful a tool you think intelligence is, and I think rationalists, and most of us to be honest, overestimate it.
Rationalists generally prescribe a 'Bayesian' world-view, which can extract useful information out of a chaotic, too-complex-to-model world.
Regarding the power of intelligence, it's generally considered to be synonymous with optimization in the rationalist crowd. It's not about being all geeky and axiomatic, but using all information and tools available for optimization towards outcomes one wants
> I feel that the main issue with the rationalist crowd is that they live too much in the space of rationality, intelligence and abstractions, but not enough in reality.
This seems like your idea of what the rationalist crowd is rather than what they actually are.
It would be highly irrational to deny or ignore reality, including the influence of emotions, irrational humans, chaotic systems, etc.
So I must ask: what is your evidence/basis for these claims?
> It would be highly irrational to deny or ignore reality, including the influence of emotions, irrational humans, chaotic systems, etc.
Well, yes, but being a rationalist does not make one rational. It just makes you part of a group, and you show membership to such a group by applying a very specific brand of rationality: using the "right" words, the "right" ideas, the "right" way. A lot of people fetishize cold, hard logic and would rather hold all emotion in contempt than do the work of understanding why it exists and what purpose it serves.
> So I must ask: what is your evidence/basis for these claims?
It's not a monolithic community, so you'll see more debate and disagreement than in a lot of other groups. So what they "actually are" is many things.
But there's often a certain ungrounded "vibe" to the conversation there. It's a breeding ground for ideas and thought experiments that I would generously qualify as dubious. Stuff like Roko's Basilisk, Pascal's mugging, AI boxing roleplay, precommitment, time loops, whole universe simulation, recursive self-improvement, a superintelligence converting the entire universe to paperclips. One of the community's most well-known outputs is a 1000-page Harry Potter fanfiction which I can only describe as a fantasy of solving everything with big brains (it's weird, but it's fiction, so whatever floats your boat).
It's the kind of thinking that makes the most sense in a smooth mathematical vision of the world, because mathematical objects are the kind that admit exponentials, extrapolations and infinities. But if you work with physical reality enough it becomes clear that it's a hopelessly messy thing that will never abide by your best laid plans. When e.g. you ponder how a superintelligent AI could escape from containment by simulating the gatekeeper's mind and say precisely what would make them free it, or threaten to torture a thousand copies of the gatekeeper unless freed, part of you is going to clock that as weird nonsense even though you are not able to explain precisely what's wrong with the thought experiment. I think a lot of rationalists either lack this mental "sanity check" or don't trust it, which lets their mind drift into a weird space.
> A lot of people fetishize cold, hard logic and would rather hold all emotion in contempt than do the work of understanding why it exists and what purpose it serves.
In the rationalist community we're discussing? Is that what you're claiming here?
> But there's often a certain ungrounded "vibe" to the conversation there.
This really sounds like evidence for my claim, that what you're saying is based on your feelings towards that community.
> One of the community's most well-known outputs is a 1000-page Harry Potter fanfiction which I can only describe as a fantasy of solving everything with big brains (it's weird, but it's fiction, so whatever floats your boat).
The point of that story is to make technical concepts more accessible and easier to ingest [0], not necessarily to make a point by itself. Framing it as a weird fantasy is disingenuous.
> part of you is going to clock that as weird nonsense even though you are not able to explain precisely what's wrong with the thought experiment.
This is again, feelings presented as evidence. The fallacy of appealing to emotion ("It must be false because it feels wrong"). Thought experiments are primarily meant to provoke thought, not as reliable predictions of reality. People are free to indicate where any thought experiment is lacking using well reasoned arguments. "It feels wrong" can be a great start for that, but never a good end.
It's not presented as evidence. I was going for the "basis" part of your ask for evidence/basis, which is definitionally looser: elaborating on my thoughts and the kind of content I've seen that made me think this.
Let me put it this way: I have spent hundred of hours reading rationalist and rationalist-adjacent posts on LessWrong, SSC, HN, Reddit; the entirety of HPMOR, a decade ago; books like Superintelligence, The Age of Em; the LessWrong IRC channel, a while back; I have reasoned arguments about pretty much all of it, including the stuff I consider to be ungrounded. What I don't have is an entire day to collect the material and lay it all out to you in a 5000 word dissertation. Do you see what I mean?
So, yeah, it's a "feeling" -- it's the "feeling" I ended up with after spending all that time engaging with the material, thinking and arguing with people, which is why I trust it -- I have worked very hard to develop this intuition. And perhaps I could pull from it to craft an argument you would find compelling, with all the necessary receipts... but you must understand that this would be a massive time sink and I'm not going to do it. It's not a reasonable ask in the context of an HN thread. To a point you just have to trust me, or not trust me on this. I don't expect you to. It's fine! Not every thread can have a conclusion.
> People are free to indicate where any thought experiment is lacking using well reasoned arguments. "It feels wrong" can be a great start for that, but never a good end.
It's triage. The better your intuition is about whether an idea will lead somewhere, the less time you will spend pursuing dead ends. In an ideal world, you dismiss thought experiments will well-reasoned arguments, but the reality is that there is an opportunity cost to it. Using "feelings" to end some conversations is unfortunately a pragmatic necessity.
> To a point you just have to trust me, or not trust me on this. I don't expect you to. It's fine! Not every thread can have a conclusion.
It's clear how you got to your opinions. I see no reason why anyone that did not already share them would do after your comments, though. Some examples or bits of evidence would have worked towards convincing readers.
> but the reality is that there is an opportunity cost to it. Using "feelings" to end some conversations is unfortunately a pragmatic necessity.
The latter is something completely different than dismissing thought experiments based on feelings after having "pondered" and tried to rationally dismiss them. One of the key tenets of rationalism is exactly that feelings and intuition can be incredibly misleading. It's fine to have them and see value in them, but thinking they are reliable when you cannot rationally support them is nothing more than gambling.
If more than 0.05% of the rationalist and AI safety community were Zizians this argument might be worth considering for like ten seconds before dismissing it.
Every major political party has violent off-shoots, as does every major religion. If we're writing off rationalists because of a half-dozen people, we're writing off the vast majority of the world for participating in even worse guilt-by-association.
I agree, and it's a particular shame in this instance, because what is startling about the HF incident - to me, anyway - isn't the degree of intelligence the agents exhibited but their persistence. I tend to believe that LLM architecture is not capable of producing a "superintelligence" in the way the LessWrong crowd defines that concept, but the combination of infinite stamina and infinite persistence is enough to cause some serious problems, even if intelligence plateaus right now.
When the "LessWrong crowd" talks about dangers of AI, they don't assume a particular form of intelligence or method to achieve it. They talk about the danger of optimization processes, i.e. "find X which minimize Y(X)" itself can be dangerous, even more so if X is a sequence of actions. "infinite stamina and infinite persistence" is one of possible forms of superintelligence in Bostrom's _Superintelligence_.
in a way you could criticize them as being idealist instead of materialist. like you said they think ideas are more important than material substrates.
most of the time brilliant human ideas are arrived at near simultaneously by multiple people because thats the affordance of technology and social and scientific development
its true ai will have better working memory and will have read more books than a person, but i don't think thats insurmountable at the frontier, which will develop slower than pure thought, since real materials need to be moved and manipulated. whoever holds the guns holds the power. the danger is giving ai effective cotrol of industrial and security processes, then it can fuck things up
thats the stuff of revolutions. when production was controlled by capitalists more than lords, the lords got overthrown. in many countries peasants and workers overthrew capitalists because they control production. hence much effort in us buisness goes into repressing revolutions. ai will be a vector the business people give control because they think it is more friendly to their interests than human workers, but they may lose either way.
To a degree you might say that they live too much in the space of the abstract and not enough in the concrete, or too much in the general and not enough in the specific.
I think that's a fair criticism a lot of the time. Most of the time we have precedent that can guide our actions well. Trying to reason everything out from first principles can be wasteful navel gazing when we've collectively seen the movie a million times.
Focusing on reality and specifics though is what causes people to say there's no global warming because December is cold.
It is sometimes possible to see a trajectory that has never happened before though and that's when you need the autists.
> The first ~1.4C of warming coincided with the greatest surge in human development, poverty reduction, and technological progress in history. For most of it, we didn't even realize the climate was warming... To me, this suggests that economic development massively outweighed any negative impact that warming might have caused.
It may also be a case of borrowing from the future. It's kind of like blowing your yearly budget in a month, it'll be spectacular until the equally spectacular crash -- and if you're unclear what your budget is in the first place, you won't see it coming either.
I mean, it's not like we're sophisticated about it. We get a glut of energy, we burn the energy as fast as we can. This provides incredible benefits because energy is basically magic, but it's not a strategy. It never has been. We don't act the way we do because we know what we're doing, we act that way because it's compulsive. Anyone who compulsively drives at full throttle in a straight line without looking ahead is living in a perpetually shrinking sliver of probability that the road is infinitely straight. They are bound to crash eventually.
I think a better angle is that LLMs are "just" token predictors in the same way that we are "just" made out of atoms. A lot of people do not believe consciousness can be material; they don't believe that you could build conscious beings just out of normal matter, without an immaterial "soul". Whereas if you think that physical matter is a lot, actually, and you can build a nearly infinite variety of things out of these building blocks... well, why not consciousness?
Predicting the next token is an immensely powerful thing. You can encode basically anything into a sequence of tokens, any configuration of matter. If you have a perfect next token predictor, you can predict basically anything. You can simulate an entire universe. It's Turing-complete. It sounds diminutive, but it's a general paradigm. Given everything this paradigm appears to be capable of, well, why not consciousness, really?
If you presuppose certain key aspects of materialism, non-dualism, functionalism, objectivism, perhaps absolute determinism... and then layer on a good general grasp of neurobiology, chemistry, cosmology, physics, computer science, evolutionary biology, and psychology... I think the conclusions you have stated are inevitable.
Humans are "just" atoms. Humans are "just" the sum of trillions of living cells going about automatic/reactionary processes, culminating in a buzzing cacophony of activity that gives rise to what we perceive as consciousness and free will.
If you plot the three primary colors with all possible combinations of only two of them, you can connect them all neatly using a circle, and distance on that circle does tell us something about how visually distinct colors are. It isn't much more profound than that.
One of the colors on the circle, magenta, has no corresponding pure wavelength, because in order to see magenta, your red and blue cones must be activated, but not the green cones -- you need two frequency peaks to do this. If we had four different types of cones, like many birds, there would be many more non-spectral fully saturated colors and you'd need something like a hypercube to represent them.
I wish we had a UI paradigm where whatever is underneath your cursor is not allowed to change. If you hover a button, that button must remain right there for you to click, if you hover some text, it must remain there for you to select. Browser and OS-enforced, ideally.
We can debate the specifics, maybe it's only in effect for X seconds after you stop moving the cursor, maybe it creates a 100px diameter disk of stability, I don't know. Just let me interact with the stuff I see.
Oh, that UI paradigm already exist. It's just that browser makers are too busy with things like webasm, webgpu, or allowing pages to launch location/notification popups at users, so they don't have time to fix fundamental flaws.
I don't think it does? Popups are one thing, but the very possibility of dynamic content entails that stuff will shift on the page. Sometimes it's just the nature of the thing, like a chatroom or a notification stream -- it'll shift as new content arrives, which is what I want most of the time, but if I'm interacting with a message, that message shouldn't shift. I'd like a general solution to this. Does that exist?
Devil's advocate, the worst-case scenario is really bad, and would probably be common: A random button your pointer is unintentionally hovering on stays put while the rest of the page wildly formats around it, super-janky UX.
Yeah, it would need to be designed and tested carefully. Rather than formatting around things, I would probably be fine with something simple like overlaying the previous layout in a small disk area under the cursor. So if a button shifts down, you'd see it shift down, but there would be a copy of it still under your cursor, which would fade on its own after a short period.
Alternatively, we could disable clicking/selecting anything that wasn't visible for at least 200 or 300 ms at that location. So if you try to click on a button and it's shifted down or something pops up in front, and you end up clicking on that instead, your cursor would just blink red.
As someone similarly frustrated by these UI shenanigans, I’m impressed with how well thought-through your idea is. It sounds reasonable! Probably not that hard to implement either. Could it be a mere vibe-coding session away from existence?
While this may look like shit, I don't actually think this is all that bad a scenario. It's a better worst-case scenario than the 'button moved right into/away from my cursor' scenario.
And if a button has appeared on the screen in the last half second or so, assume that any attempt to click it was unintentional (unless it appeared due to an explicit action by the user).
Similarly, if a button has recently moved, wait a little while before activating it. (Better yet, arrange for things not to move around as the page is loading.)
Similarly, the UI must not interrupt me when I’m typing into a text field!
I’ve lost count of the number of modal dialogs I’ve “accepted” because they popped up out of nowhere milliseconds before I pressed enter coincidentally.
I'm not saying degrowth is feasible, mind you. I agree that it probably isn't. But the fact we can't do something doesn't mean the alternatives must be good. It might just mean we're cooked.
reply