> We have now seen that even the rumor of someone working on a problem can trigger a massive amount of AI-powered effort to flatten it before the original research project has time to reach its full potential.
In other words, the "right" people need to solve it: the mathematicians who made it their job and not the people working to push AI models forward?
Struggling to understand how a solution to a millennium problem like this isn't a net positive. Presumably Open AI employs mathematicians in these efforts anyway. And I can think of far worse uses of the AI compute resources.
If there are no incentives for mathematicians to work on and share progress in tough problems because they will get scooped, then they will end up quitting and in net we will see less progress overall.
Some people have to work a thousand times harder than someone else for 1/1000th the payoff due to their starting point. That's just life. But that doesn't reduce the impact your choices have on your own life. With a few extreme exceptions you can choose to change your environment. It's just varying degrees of terrifying/uncomfortable/dangerous.
That still does not imply that ">it's quite likely that nearly all of your problems are the result of your choices, either indirectly through the environment you live in or directly through the actions you take".
You changing your environment to void problems plenty of times means "abandoning your kids".
You may even have greates starting point ever and still have externally imposed diseases, injuries and problems.
I could be mistaken but my interpretation is that "studying for your own good" is being contrasted with "studying so you don't fail". If you actually fail students, they'll have a second motivation beyond trusting some authority figure's advice. Namely consequences.
At the time it was published eye tracking would have come across as rather over the top (IMO) despite being a predictable outcome if you gave it some thought. Plus it would have necessitated adding and explaining cheap AI analysis, all the follow on plot impacts that would have, and would have ruined that scene as written. So I'd say that conveniently omitting it was a wise choice.
Can be easily handwaved away if you assume some toothless anti-surveillance legislation outlaws employers from engaging in eye tracking while failing to ban all the other stuff. That's not even an unexpected real world development at this point.
> It has been over a decade since any ordinary off-the-shelf closed-source software was meaningfully obscured from serious adversaries.
Probably goes without saying but the last line of defense is not deploying your software publicly and instead relying on server-client architectures to do anything. Maybe this will be more common as vulnerabilities are more easily detected and exploited. Of course its not always feasible.
It has been annoying seeing my (proguard obfuscated) game client binaries decompiled and published on github many times over the last 11 years. Only the undeployed server code has remained private.
Interestingly I didn't have a problem with adversaries reverse engineering my network protocols until I was updating them less frequently than weekly. LLM assisted adversaries could probably keep up with that now too.
Honestly, I can't really imagine how this would work at all?
I could see how, given enough data, you'd be able to infer the intended logic of the server and reimplement something that's compatible (I've done this myself with Wireshark + USB devices in the past).
But how would could you reason about specific vulnerabilities in remote code just from a set of requests and responses?
Because while you could get something that drives a dumb interface, by moving the work and data to the server it's not available for the emulation software to use.
If the contract is well defined, the LLM can infer what it's purpose is, implementation, possibly even your secret sauce. There is no software moat anymore.
yes this is what i was trying to say. its quite common on older client-server games to do this sort of thing. powerful ai models will just make the work to recreate/emulate servers faster.
Except that emulating what is seen is surprisingly useful to find attack vectors. As a single deeper datapoint, one can look at more than just baseline behavior and delve into timing details to further refine implementation guesses.
There is a reason that pilots get basically told the ins and outs of a specific plane. Imagine the outrage if people need to do month long training for a specific car just to be able to drive it (and not just a general "here is how cars roughly work and the laws of the road").
Airline pilots aren't supposed to take a nap, and there are occasionally articles about the various things that have gone wrong because the pilots weren't paying attention.
> You would not start with "The car is functional [...]"
Nope, and a human might not respond with "drive". They would want to know why you are asking the question in the first place, since the question implies something hasn't been specified or that you have some motivation beyond a legitimate answer to your question (in this case, it was tricking an LLM).
Why the LLM doesn't respond "drive..?" I can't say for sure, but maybe it's been trained to be polite.
In other words, the "right" people need to solve it: the mathematicians who made it their job and not the people working to push AI models forward?
Struggling to understand how a solution to a millennium problem like this isn't a net positive. Presumably Open AI employs mathematicians in these efforts anyway. And I can think of far worse uses of the AI compute resources.