Because their leaderships have defunded / fired all ethics researchers looking into their ongoing crimes and failures in favor of a bs "AI safety" narrative based around a creepypasta about an AI monster time traveling backwards to torture their staff for not maximizing shareholder value.
You're insulting the intelligence of everyone on this website by even attempting this argument. The handful of upset elections by young socialists who want to decrease the political hegemony of billionaires does not amount to evidence that the political hegemony of billionaires isn't real.
The enshittification model, but with an extra step because they actually have to employ labor.
First they exploit their own gig workers, to make an unsustainably cheap product and corner users. Then they exploit users, to monopolize distribution and corner restaurants. Then they exploit restaurants, threatening to cut off distribution entirely if they don't get an increasingly large share of profits.
Came here to post this. AI rationalists love thinking about paperclip maximizers, but don't seem to care when it turns their own codebase into paperclips. Or to rephrase, turns their whole engineering org into meat proxies, slowing down engineering productivity in the long term because understanding is drained out of the staff and flushed down the drain every time they close a Claude Code tab.
How is this different from arguing that Microsoft Clippy was RSI? An AI tool being involved in the process of work can't be the bar for RSI.
I don't think there can be a coherent definition of RSI unless people lay out their theory for how intelligence scales. LLM-assisted coding is great but respectfully optimizing pytorch features or whatever is not gonna lead to exponential improvements. That approach to scaling diminished years ago, leading all the labs to switch to reasoning.
Now it seems reasoning is also yielding diminishing returns, so all the labs are pivoting to specializing in particular fields like math / infosec / biology. They're improving due to accessing new proprietary training data and doing RL with human experts. Again I don't really see any amount of "AI research interns" leading to an exponential improvement to this strategy, they're not the bottleneck in the first place.
>Now it seems reasoning is also yielding diminishing returns
Is the diminishing returns in the room with us?
>so all the labs are pivoting to specializing in particular fields like math / infosec / biology.
They're not pivoting to anything. The goal has always been creating a machine that could automate all or nearly all human work. They're just coming along on that mission.
As for RSI...I think the term is a bit odd in the modern context. It was created at a time when conventional wisdom was that generally intelligent machines would be these logic automatons that could "alter their own code". Instead we have massive neural networks that take months to train.
In this paradigm, the ways a LLM could "improve itself" would be altering its own weights directly or creating and training better, vastly more efficient architectures for the next generation of models.
The former is probably not happening but the latter is possible.
Yes, diminishing returns. Not overall, they've still been able to create more intelligent models even up to today. But the strategy for scaling that intelligence has shifted. From the initial ChatGPT release to GPT-4.1, they were basically scaling up compute training compute / model size. Then 4.5 flopped, while o1 demonstrated that gains could continue by reasoning (scaling up compute at inference time). o1 is now the ancestor of all their flagship models from GPT-5 on.
This is why I'm trying so hard to drill down on the theory of scaling, and not just talk about improvement in general, hand-wavy terms. If the bottleneck of current scaling strategies is training data, or something fundamental about the model architecture, then just throwing more harnessed chatbots at it won't lead to an exponential increase in performance.
Now you could argue that the AI we have now will help us find that change in architecture, and I would agree. But that means we're firmly outside the singularity for the time being, and what people are in fact talking about is a hypothetical.
>Then 4.5 flopped, while o1 demonstrated that gains could continue by reasoning (scaling up compute at inference time). o1 is now the ancestor of all their flagship models from GPT-5 on.
That's not quite right. They are still scaling model size and have had several new base pre-trains, just nothing so big as 4.5 (as far as we're aware). o1/4o has not been the base for some time now.
Data is obviously a bottleneck for some regimes and LLMs will have to get their hands dirty experimenting but it doesn't look like an insurmountable wall either.
> Now it seems reasoning is also yielding diminishing returns
Not true. On the contrary, LLMs are developing faster than predicted. They were expected to solve a Millennium Prize by 2030... and here we are in 2026. Release cycles are getting faster. Just compare the most recent GPT or Claude with what they were an year ago.
> How is this different from arguing that Microsoft Clippy was RSI?
We can argue about semantics, but that's not really the point. The point is that what started now - which no doubt is in its infancy - will result in full autonomy quite soon (they project an year or so), with the risk of RSI causing agent development to slip (long term) outside human cognitive control/capacity.
Again, can you lay out your theory for how intelligence scales? You're using a lot of terms like "full autonomy" without definitions. Why do you think that just throwing more harnessed LLMs at (something?) will lead to an increase rate of improvement?
I feel like I laid out several cases where other things were the limiting factor on improvement and more agents wouldn't have helped, and I didn't get a response to those cases.
What "they project" (the labs) is of minor interest to me. Aside from their incentives and track record of lying, in recent months they are laying out a story that is pretty much just the plot of Terminator, and directly referencing rationalist beliefs that were published long before LLMs even existed.
OpenAI/Anthropic have never pitched themselves as a replacement for farmers. They do explicitly say that they're going to cause significant job loss in knowledge work sectors all the time.
Right - I'm saying that you can greatly improve productivity / reduce employment while still having humans in the loop. We've already seen it happen with farming, from 1900 -> present.
A chatbot for cancer researchers to talk to is worth single-digit billions at most. Anthropic is already valued at over a trillion dollars, on the premise that they can replace the majority of jobs in most knowledge industries. All the announcements about hacking / math problems / biological science are meant to create the impression that that strategy works and is repeatable across industries.
Cancer research is a lot harder for LLMs than math millennium problems though, because there is no fast feedback loop to iterate on. Even if you have a really good idea based on a solid theoretical insight, doing the experiments using in-vitro/mice/monkeys/humans can take years or even decades. I have no doubt that AI will help find new avenues that boost certain parts of research in these fields, but I don't see a potential for a drastic change until we at the very least give LLMs a direct way to interact with lab equipment and train them using RL on it.
Yes, but initial discovery of molecules and novel mechanisms is massive. That was the last generational change in modern drug research was the movement to high throughput screening, going from the ability to screen 10's of molecules to hundreds of thousands to find 'hits'. Better and more focused models, especially ones trained internally at big pharma companies will accelerate that portion of the pipeline, or increase the hit rate of successful compounds. Several companies are already taking this approach like Novo has been. There are other more early stage companies like Recursion and others that are doing the same thing. They are more tech companies than traditional wet lab companies.
Sorry, yes I agree 100%. I don't agree with their narrative, I was just explaining it. I think it's fraudulent and based on science fiction and will lead to a significant economic crisis.
Our lab, located in the Bay Area, looks like a typical molecular biology lab. We do research that involves only the lower-levels of the biosafety risk level (BSL-1 and BSL-2) and we do not handle pathogens that can infect humans. All of the lab work is performed by human scientists. Although we’ve experimented with using AI to accelerate lab work with initiatives like the Model Hardware Standard, this approach is less conducive to the sort of ad hoc workflows that are involved in our molecular biology research.
Automated labs are far more challenging to build and run than most people appreciate. If I wanted to make progress quickly in discovery science, I would find good lab techs before building automated labs.
Every perpetrator of genocide throughout history has said exactly this. If we don't do it to them, they will do it to us! It is nonsense in every case, and only speaks to the genocidal worldview of those who believe it.
reply