There's a book, "Impro", about a theatric studio that practiced improvised sketches. One thing they struggled with was a normal conversation. People have them all the time yet somehow artists could not find the right approach.
One day the head of the studio (the book author) had an insight. She asked the actors to drive the conversation so that they always bump the status of themselves or the other party in any direction, but only a little.
This did the magic. Not only the actors understood her perfectly, the conversation started to flow and could go endlessly while staying amusing and captivating for spectators. (Look up the book for examples.)
After they discovered that secret they started seeing the pattern (status games) everywhere up to the point they asked themselves whether there is anything else in human communication.
This was a British studio, the sketches were mostly comedy.
> After they discovered that secret they started seeing the pattern (status games) everywhere up to the point they asked themselves whether there is anything else in human communication.
Status truly is your god, whether you would admit so or not.”
i have read your comment in full, multiple times, and I still don't understand what you re saying.
> that they always bump the status of themselves or the other party in any direction.
what? English non native so that might be the reason
Every point they add makes themselves or allies look better (sometimes just more interesting, sometimes more humble) or their enemies look worse. Usually themselves better, at least me; it’s inevitable, even explaining this I notice I’m doing it.
The key for the improv actors is doing it with intention. To decide “what is the next thing I should say in this conversation” they think “how do I make my fictional self look better, or my fictional conversational partner look worse, to the audience”? And that gets them to natural-sounding phrases.
Is that some fundamental behavior of humans confirmed by psychology? I've never heard of this before and most people I encounter don't do this. Of course there is selfishness and altruism, but it seems this is referring to something else?
The thing that GP and me (briefly[1]) probably struggled with was the meaning of "status". My first connection was "status" like in "status update", which almost makes sense in the context of conversation. Whereas what you meant is a "social status".
Social status, not state as in health or other ``facts'', though?
I barely comprehend social etiquette on meta level of british people, so it is not just a language problem if the word used was a little too broad.
But it could work that way and I rarely noticed, since people are supposed to talk over the weather if starting smalltalk. Not necessarily about themselves.
People don’t understand just how important Lean and Mathlib are to this whole endeavor, so they mistakenly attribute all these accomplishments to “AI”.
Intelligence like consciousness is a word to describe patterns of processes tending to optimize toward some goal set under the constraints of environment and with sensory feedback. Humans goal is to reproduce. Llms goal is to predict best next token. Human feedback is sensory mediated. Llm feedback is prompting on outputs including from tools such as lean. If RH was provable it is not surprising either such system could prove it in principle. If a sound proof required an intricate path or vast contextual memory to assemble it is not surprising a machine proved it before a human. However I don't think this is generalizable. I don't think intelligence is a scalar in that the proof shows the machine is more intelligent than the human. Rather it is qualitative. The machine was structured so that the path was attainable to it. This is intelligence in the particular case but not general intelligence if such a thing exists. Now what would point more toward general intelligence is a machine that can compress proofs or truths better than humans. If we start seeing short proofs of FLT or RH then that will be interesting. I am skeptical with the current technology. The tendency of LLMs seems to be to explode complexity not condense things to essential elements or minimal description lengths.
My understanding is that there is no central, agreed, library of lean formulations of hypothese+axioms for big open problems. So it's not like, when proving one of these you necessarily just take the lean-encoded problem off the shelf and add the "i proved it" parts.
First step seems to be you encode the problem according to your whims. So there's some checking to be done in terms of "does this actually encode the problem?"
For pro mathematicians I wouldn't expect that that's where they run into trouble in things like this, but still needs a look-over.
Claude's proof just uses Mathlib's formalization of Riemann's Hypothesis. It's the closest thing you can find to an "agreed" one. It's not like that Claude tried to formalize it from human language from scratch.
Often (most?) of the time "solving" a hard problem is partly taking it apart into separate pieces and phrasing those pieces in various novel ways that allow one to tackle them individually and then at the end it all magically falls into place (in this case after 900 pages).
If we can trust payroll to a database instead of insisting someone do all the calculations by hand there is no reason to expect other forms of symbolic manipulation to be vastly different.
We can and should trust computers for the straightforward, boring parts. But there's a huge gulf between the people who think, "how hard could payroll be?" and actually making a system that handles all the interacting complexity of real-world problems.
Computers have no more or less a finite resolution problem than humans - it just seems to be far larger than what is practically needed in both cases. I.e. problems can be encoded symbically or numerically and the maximum size of each (including the numerical precision of the latter) is bound by physical constraints. But just because computers use fixed sized numerical types does not mean they are unable to handle working with exact symbolic values or objects we use in math.
Regarding the Pentium bug, I think one has to agree when it stopped being up to the human introduction of the bug. I.e. sure, many think we have designed faulty CPUs before. Some feel computers have designed faulty computers, others feel it still comes back to the human definitions. At some point though, most seem to call human mistakes human mistakes rather than where they are derived from prior to humans. I just don’t think we’ve all reached agreement of when that boundary definition is is for mistakes from the computer.
If I don't get my paycheck, I immediately notice and can complain, and a human will be able to rectify whatever issue caused it. If I literally never got a paycheck again, we have decidedly non-mathematical ways of dealing with it (a legal system).
I don't see the comparison here. Detecting issues in a 17 million line proof is not remotely trivial, if we accidentally build a bunch of math on a failed proof, we don't have a retroactive way of getting all of that time and effort back.
I'm not saying that automated proofs like this aren't worthwhile, but the parent comment did say "functionally equivalent". The level of trust that's needed in the people who used the system to produce the output is a pretty clear functional difference in my mind between payroll and huge generated proofs. I don't think I need as much trust in someone who produces an automated proof as someone who claims that god told them something directly (since that's a lot harder to believe in general), but I also don't have a particularly large amount of trust in the average employee of Anthropic making a bold claim either.
There have always been a large number of people in the world (and still are!) that believe any number of things on the basis on faith rather than evidence, and it's not crazy to draw a parallel between "that religious authority says so" and "that Anthropic employee says so"; for the overwhelming majority of people in the world, the claims are essentially both impossible to falsify, so at the end of the day we are still relying on trust (whether from that authority directly or from a consensus of other people we trust).
So is natural deduction when you treat quoted expressions as first class citizens.
The difficulty of doing proofs in mathematics is largely due to the fact that mathematicians refused to treat symbolic expressions as data and had to invent ever more complex paradigms to deal with them.
Lean being the final iteration of that trend.
When you add quotation and quasi quotation on top of Polish notation proofs become as mechanical as long division.
idk what you're talking about. mathematicians treat symbolic expressions as data all the time. They apply a procedure to one expression to get another expression.
If I ever get the time writing a book called the structure and interpretation of mathematical logic is on my to do list.
There are unfortunately no books on the topic currently, because very few people take syntax seriously enough. This is rather ironic as computer science is only possible because someone took syntax seriously enough to invent place value notation.
i don't think mathematicians have avoided proof assistants, being that they have been around for over 50 years (https://en.wikipedia.org/wiki/Automath). however, as to my point, i would wager most payroll systems don't express a single verification as 400kloc of SQL statements.
There are some mathematics who don't mind formal methods, but they are few and far between. A lot of them are closer to computer scientists than traditional mathematicians.
Generating a paystub isn’t a Millennium Prize Problem.
Everyone has limits to their domain knowledge that makes understanding expert material challenging. I’m guessing that at the pinnacle of any field, there might only be a handful of people who fully understand the solutions to the most complex problems.
It does not. If after careful analysis people conclude that they cannot tell if it says what it claims to say, then then it can be considered not-a-proof until someone can demonstrate in a convincing manner that it is what it claims to be.
I can say god exists because 5>3. No matter how valid my claim is that five is in-fact greater than three, if I can't convince people of the connection it's a valid proof of something, just not what I say it is.
It might be true that if the Riemann hypothesis is false, leaves cannot fall to the ground. That would mean a falling leaf demonstrates the truth of the hypothesis. Without showing people why it must be the case in a way they understand it falls short of a genuine proof.
That would make proofs subjective, and I think that is true. To many people Pythagoras has not been proved, they just believe (correctly) there is a consensus amongst those who do understand the proofs.
For a lot of people there is no difference. I'm fairly well versed in physics for instance, but I quickly lose my ability to deeply understand what's going on when quantum mechanics is introduced. I can just about grasp the analogies, but that's the limit of my understanding. So the stone tablets have become YouTube videos and God has become the likes of Feynman.
We are nowhere near that. Yes its getting harder, but i'm certain that within a year people will have this disciphered and translated into traditional math prose.
Keep in mind deciphering and validating traditional proofs is also quite hard. e.g. It took a long time to figure out if the proof for fermat's last theorem was valid
You not knowing how to interpret the results is not the same as nobody knowing how to interpret the results. Formal proofs with very dense lean implementations have been a thing for a long time, it wasn't 'god on the mountain' then, either.
I don’t think it is possible for any human to actually understand 17.9 million lines of code.
I don’t think that things humans can’t understand meaningfully advance human knowledge. Scientific progress is built on a foundation made of previous discoveries. What is there to build on here? An incomprehensible soup and a note from the computer that says “trust me”?
"I returned to civilization shortly after [the Trinity test] and went to Cornell to teach, and my first impression was a very strange one. I can't understand it anymore, but I felt very strongly then. I sat in a restaurant in New York, for example, and I looked out at the buildings and I began to think, you know, about how much the radius of the Hiroshima bomb damage was and so forth ... How far from here was 34th St? ... All those buildings, all smashed - and so on. And I would go along and I would see people building a bridge, or they'd be making a new road, and I thought, they're crazy, they just don't understand, they don't understand. Why are they making new things? It's so useless.
But, fortunately, it's been useless for about 30 years now, isn't it? So I've been wrong for 30 years about it being useless making bridges and I'm glad that those other people had the sense to go ahead."
Plan as though you're going to live forever, live as though you're going to die tomorrow. This means that you fix the leak but treat it like a precious and divine activity that you would be honored to have as your last.
I think when people set out to create a chess playing computer they thought we’d end up with something that we could deeply understand at an intuitive level. Instead, when you look at the Stockfish evaluation of a position, you do get an “explanation” but it’s really just the aggregate of a bunch of calculations, which are themselves stacked on top of the “stochastic parrot” NNUE evaluator.
This leaves us in a very unsatisfying position as creatures who want answers to “why” questions. Intuitive answers are incomplete, and complete answers are either fundamentally impossible or completely incomprehensible or both.
Feynman said something similar with regards to magnetism, but I feel that understanding a chess position is an even better example. Magnetism depends on facts about the universe which at their most basic level “just are”, whereas even a complex chess position is theoretically entirely explainable.
I hadn't seen that clip in a long time. Maybe the younger version of me was impressed by it. I understand the spirit of what he's saying. But that clip makes him seem extremely unpleasant. Presenting bits and pieces of the framework is how everybody learns. If somebody slips on ice, I'm not necessarily concerned with why ice is slippery, even though it's a logical part of the reason.
There's plenty of intelligent people who can answer a question like that without devolving into a lecture.
My undergrad is in physics and I was the electrical officer on my first ship. There were plenty of exotic things about that plant that stupified me, but it seemed like no one else was interested in those things. All the other junior officers were busy quizzing each other on the number of cylinders in the diesel engines that drove the generators, what spaces the generators were in, shit that didn't really seem to matter to me.
One afternoon underway I was standing in Central Control and the main propulsion assistant, more senior to me and studying for her engineering boards, started quizzing me about the electrical plant. And I would explain things. And she would ask why. So I would explain and she would ask why. And so on. Until we got to fundamental wave equation for electromagnetism. And she kept asking why. And at one point I realized we were at the bottom. There's nothing really below electromagnetism. It just is. And I said that. "It's one of the 4 fundamental forces. It's just a mathematical description of the way the world seems to work, and it's proven to be correct, entirely, in all cases. There other forces at different energy levels, but in the world that affects you and me, it just is. It, it just is."
And at that point I realized half the engineering department was peeking in from around the consoles. And she starting crying. And I felt so bad. Like I had said something wrong, and I still don't understand what I should have done better. That was almost 30 years ago.
So Feynman's observation that you have to be in some framework that you allow something to be true strikes me as incredibly helpful. I think the MPA must not have had that kind of framework and found herself distressed when she kept reaching for things and never getting something more to hold on to. Maybe I should have had the insight to offer her more of a lifeline.
No, I agree. What he's saying is technically correct. There has to be a certain framework in order to fully justify certain beliefs.
My point is that the answer you give depends on who you're talking to. It's okay to present a simplification or an incomplete picture. That's how everybody learns. Add a disclaimer and move on. There's a time and a place for technical correctness and, unless the interviewer was being critical/annoying prior to that recording, I would say the frustration is misplaced. There's no need to get frustrated at the interviewer because you can't meet your own standards of rigor.
Some people i've shown this to (in admiration for Feynman) have had the same reaction, but I just cannot see it. I'd be interested to know why you think he's unpleasant.
Are you a physicist? I wonder if non-physicists just misread his frustration. The question isnt about some macroscopic physics problem that decomposes clearly into something satisfying. Its a question about one of the most basic constituents of reality. A "why?" question about something so basic has no answer other than "just because". Insofar as anything can be said, it will be so abstract and technical that it wont satisfyingly address the question.
It is a little like a lay person asking a mathematician: why is 0*1 = 0, but 0+1=1 ?
Except, somewhat more extreme than that, because such basic things as 'magnetism' are introduced into all accessible physics formalisms as assumptions.
Because he projects a certain kind of frustration that's not warranted in that moment. He's being asked a technical question without any means to give a rigorous answer. I appreciate rigorous thinking, but the interviewer doesn't necessarily want a perfectly rigorous answer, nor does the average person. He's not responding to a need, he's projecting his own perfectionism.
The interviewer most likely would not have interrogated him further about the underlying properties of ice (or engaged in that entire fictitious interaction). Feynman clearly understands that there are several levels to understanding.
So just give an answer that you feel is appropriate, add a disclaimer ("this answer is hand wavy"), and move on. Why is that so hard?
The answer is, "I cannot explain that force to you. It is just, as a matter of fact, one of the fundamental forces of nature."
He's taking it as read that the reason he is giving his extended answer is to explain why he cannot answer at all.
What he could have said to a physics student is that: wave-functions of charged particles have a property, "phase". The problem is that point values of phase cannot be observed, and if you formulate the laws of physics with raw "phase" values (like a measure of mass, at 5kg), they dont work. So something must exist to "track" phase, so that every pair of measurements "cancel out" the effect of this property. That something is the EM field. Magentism is a spatial aspect of this field. Eletricity is a space-time aspect of this field. So the magnetic force arises out of a need to "conserve phase", and that phase is how a pushing force is experienced: two particles experiencing a magentic interaction take on different phases (their wavefunctions interfere such that..).
This might "feel" like an explanation to the physics student, because they will be willing to make the assumptions needed, and they will find conservation-style explanations sufficiently satisfying. ie., they can already see a wavefn description will have a free phase term. But a lay person wouldnt feel satisfied because more unknowns have been introduced, and conserving "another unknown" is no advancement in understanding.
That's the gulf which is frustrating feynman. There's nothign to be said to the interviewer other than the force has no explanation in terms of anything he's familiar with. The question is too basic. And feynman is not going to go thru a formulation of QM in terms of wavefns in 5min -- it would be an entire lecture course.
We might have to watch the whole uncut recording to understand. Perhaps Feynman thought he was talking to the interviewer rather than to you and I and the whole public.
But I have learned something very useful from this non-explanation explanation. Sometimes you have to assume stuff, and sometimes you have to dumb it down. For example, when your five year-old asks you how magnets work, you're just not going to break out Maxwell's equations. Instead you'll explain how everything is made of very tiny things called atoms, and that all atoms are like tiny magnets, but some kinds of atoms can be made to line up and then all the tiny magnet forces add up to a big magnet force. The kid'll eat it up. I know because that's roughly what my father said to me way back when. And so you'll have different explanations for different people, up to the level of your understanding.
The problem here is that not only are we missing context in that clip (had the interviewer been annoying?), but Feynman barely disguises that he can be unpleasant. He has to prep himself to tell the guy that he's just not a physicist. He has to chastise himself for being so good at this that people expect him to be able to explain anything and everything even though it's not always possible. He's kind of an ass here. But what a great answer he gives! His visible annoyance is almost necessary.
I wonder if feynmann felt like he was being used, like a circus animal. Hey Joel Robuchon how do you make the perfect scrambled eggs? Hey Einstein what's the square root of 34224452 - look at him he's just like a computer!
Please don't sneer about people on HN. If you have a substantive critique of someone's ideas or work, you can express it thoughtfully. The guidelines ask that of all of us here. https://news.ycombinator.com/newsguidelines.html
I imagine the machine proving that PPPRK beats PNBK in most configurations, but it not being understandable by any human and pretty hard to disprove except for machine.
doesn’t seem useful to humans, only to other machines
Derailing slightly, a variety of HN posts (the blogs/sites they reference) have started banning me with a 403 page from nginx, including this author. My IP reputation is supposedly fine. Does anyone have any ideas where I should look next?
- The haveibeenproxied link indicates my IP is shared (expected)
- This is stock android/chrome with a DNS blocklist of advertising domains and some invasive Samsung the things I can't fix on the device (I doubt that's the culprit given that this is a new behavior I'm observing in the outside world after 8yrs with those settings and also because it's be strange for nginx to be the thing to immediately report that trackers have to load for me to get a glimpse of the main page, especially as something I'm seeing across multiple sites, but that's not impossible).
- No VPN
- I'm a "normie" internet user. Anything remotely questionable to even the cheapest and least aligned of LLM judges and executioners happens on non-home networks.
- I see it with other devices on the same network (Chrome, Linux, MacOS, Safari, Qutebrowser, etc), which suggests to me that somehow my IP or some other shared network property is the issue. I'm curious mostly about the commonality in separate nginx installations returning a 403. That suggests there exists a common blocklist or whatnot I can inspect and fix, either because of my IP or because something in the loop is forcing ipv6-only or who knows what.
That IP shows up as an exit node in 3 separate proxy networks, one of them a malware pool. The IP really is relaying proxy traffic, just not yours. It's unfortunate, but not much you can do. One of the many reasons I hate CGNAT.
Disclosure: I work for Spur, the company behind HaveIBeenProxied.
I suspect they will sell the data to firewall companies like CloudFlare or network security companies like Cisco so they can secure their own customers that start sending out suspicious traffic. Alternatively, they could help insurers adjust cybersecurity insurance when the insuree's network sends lots of suspicious traffic.
I've been getting by at home on a T-mobile hotspot, and haveibeenproxied now reports Kookeey and Shifter residential proxies for my IPv4 shared with God knows how many other customers.
Might moving to a T-mo 5G home box help? Guessing that an IPv4 would still be shared with other people's TVs ...
I checked a small number of pages from the history of a defunct music festival site on WayBackMachine two days ago, on my phone, but found that the site needed a desktop to work well.
But WayBackMachine has been 419ing (from nginx) my requests from my desktop these couple of days, when I’ve tried. Safari on a boring macOS installation. Same network (I mostly don’t use the cell service on my phone, when at home). Haveibeenproxied returns all clear.
Maybe an ipv6 vs ipv4 thing? I have custom dns on the desktop and I think it only resolves ipv4 (maybe?) while my phone is using my upstream’s default DNS and probably prefers ipv6.
It has occurred to me that the cost of shit like this, and of an extra 1-2 second periodically on site loads across the web to block bots (this is WAY up the last couple years) might go surprisingly far toward negating the time savings provided by LLMs…
Depends on what you do? The site loads fine to me and you gave no indication what your originating IP space is being used for. As a residential user without anything but "normie" internet usage (reading blogs, watching yt) I experience no interuption. Are you a "normal" residential user as well? Then you might need to bring it up with your isp
Check if something on your network is running malware. You might be part of a DDoS botnet, and other services might be running something like fail2ban.
The same machines that proved the GRH are happily prancing around the interwebs bloating everyone’s server bill, so you put up a bot wall with challenges that give you a lot of false positives, but also peace of mind that you won’t pay 200$ more in hosting bills this month.
Funny enough this put things into perspective for me, it's like when breakthroughs happened after the first transistor based computers, yet for decades after those, we had nothing and this leads me to believe, that we might settle into problems that are still quite hard for even llms, and not everyone can even produce stuffs like this even if they had the same compute that alpoge had.
Aren't we approaching the limits of physics, with increasingly diminishing returns each generation? I remember reading an article about quantum effects at these scales, which make everything much harder. I mean, for example, even if we double the speed at which we can travel every year, you will never get to Alpha Centauri in your lifetime
Chips are only 2D, and we're far from the landauer limit, and the landauer limit only applies to non-reversible computations anyway.
If we could assemble matter atom by atom at scale we could get to some dirty diamond that fits in your hand, outclasses the world's top supercomputers and runs off 10W of sunlight.
Keeping with the Interstellar framing, the key is not only in faster ships (lightspeed) but in shrinking distance (warping space-time). For LLMs, that'd mean efficient model architectures on efficient chips.
Right. I’m sure at some point in the past more capital was spent on designing better airplanes than on building them & the infrastructure to support them.
We still spend money on designing better airplanes, but the two lines on that graph crossed long ago.
> Airliners are still subsonic after a 100 years of aviation
That is a deliberate decision motivated primarily by economics and secondarily by environmental concerns. In fact airliners today are somewhat slower than they were a few decades ago because the lower speeds are more fuel efficient. We certainly could build fleets of supersonic airliners if it made sense to do so, it is not as if we have no idea how to do it and in fact we built supersonic passenger jets once upon a time.
Your example contradicts your point. Airliners are still subsonic after an extended period of time, and hence based on that past performance it's a good indicator (although not definitive) that they will remain subsonic for an extended period into the future.
In general, past performance is actually indicative of future results. It's not definitive, and some phenomenon are random, but the trajectory of technology over time is not some random fluke that just has coincidentally trended by random chance.
>Your example contradicts your point. Airliners are still subsonic after an extended period of time, and hence based on that past performance it's a good indicator (although not definitive) that they will remain subsonic for an extended period into the future.
There was a period in aviation development when it was widely assumed that speed will continue to rise. Just like in the development of CPUs there was such a period, quite recently.
a single company was able to send more satellites to space in the last 4-5 years than all the things sent combined, across all countries since we started putting stuff into space in 1957. Saying "Airliners are still subsonic" is like saying that cars still are still not going much past 120/130 km/h thus we're technologically stuck, in a world where jet engines exist.
But along other axes they have also improved tremendously, no? Ie rather than just time to destination, they have greatly improved efficiency (energy spent per unit of work) and throughput/parallelism (ie simultaneous flights in the air) right?
Yes, but these axes are also not showing an unlimited growth. There is no expectation that either speed, fuel efficiency, ticket price, or total number of flights will improve by an order of magnitude in the foreseeable future, unless there's a technological breakthrough (Nuclear turbines? Artificial photosynthesis making kerosene free? 3D printing the entire plane? These are currently all sci-fi ideas).
Sure you're going to get a lot of math problems solved, but food distribution is still going to be stuck by warlords and oligarchies.
Think of it like highway designers and the "one more lane" gridlock delusion. In this case, it's more will smith eating speghetti than cheap, low cost cancer cures.
> Airliners are still subsonic after a 100 years of aviation.
Yeah, but that's a choice made for very good reasons:
- Noise reduction
- Fuel efficiency
- Reduced pollution and lower environment impact
- Reduced cost of materials, equipment, and maintenance
- Passenger affordability
It's not that we can't make supersonic airliners or that the fact we don’t illustrates a lack of progress or performance: it's that it doesn't make economic or environmental sense to.
Gulfstream and Sukhoi even had a partnership to develop a supersonic business jet (which, if you look at the mockups, might be one of the coolest looking planes ever conceived) but they gave up after spending a fairly enormous amount of money already because it was going to cost an absolute fortune to bring to market... and there was no real market for it.
EDIT: the responses I’m getting are a bit weird. People are putting words in my mouth but all I was actually doing was pointing out the the example used wasn’t a great illustration of the point the person I was responding to was making. This isn’t Reddit so maybe we could reign it in a bit, please?
That's kinda the point I think ? It's not like we can't turn half of global energy production or GDP to compute - it's more about does it make sense to. As much as I'm getting value out of this technology daily - the level of investment made to get this far is astronomical compared to the value I'm seeing created, even less captured by these companies. And because the progress is so good in some domains it's easy to believe that these things are magically reaching some intelligence generalization threshold. But then you leave the training distribution and you get disturbed by how poor the performance is and how obviously equivalent things don't automatically transfer at all. Given the recent market trends I would not be surprised we see a correction in level of compute investment going forward.
The universe seems to be set up such that it wants entropy to increase. Any living thing that doesn't profit from converting low entropy energy to high entropy energy dies.
So the question is whether more intelligence will lead to faster increase in entropy. My guess is it will. Whatever recent market trends are, they are irrelevant. Whether the AI can or can't do x thing is irrelevant. What matters is whether the companies that make AI profit from making AI, and if you look closely at what's going on it sure seems like they do. If you think otherwise it may be because you're reading headlines and propaganda articles and not digging into the fundamentals.
So maybe we have a temporary correction/slow down, because these companies took on too much leverage and grew artificially too fast. That's very possible. In the big picture long term it will be a blip. The universe seems biased to increasing the volume of intelligence and I have never heard a compelling case for why this wouldn't continue for a long long time.
>It's not like we can't turn half of global energy production or GDP to compute
That depends which part of 'we' is running the economy. Just imagine an economy that makes mass massive amounts of money but doesn't align with human needs.
I can argue all of the reasons are valid for keeping compute at a certain speed range, or power consumption envelope.
It's not we can't make compute much more denser, there's a new standard informally called as megawatt racks. But deploying them at gigantic scales bring in massive problems as well, so we can keep total energy envelope at a fixed range.
Rather, nobody wants them at the price point necessary to make them economically viable. In the context of tech, the cost to develop a next-gen fab doubles every four years; absent some major breakthrough, we have about 20 years before even the most advanced economies balk at spending $1T, $2T, $4T, $8T on a single fab, causing hardware to settle down like airline speeds have.
the best computing power used to be located at the universities and national laboratories. Now you need access - employee or otherwise - to a nasdaq listed startup. Hard not to think it's the shareholders who benefit in the short term in this new alignment.
I wonder he good we will be at not understanding anything to an ever increasing degree, with absolutely no hope of ever catching up. We will be super bad at absolutely everything within sciences and engineering and completely unable to contribute anything of substance, outside our own moods and quirks.
Can humans be okay with that? Can we be happy cats?
> Can humans be okay with that? Can we be happy cats?
Sometime in your youth you decided what was important to you, and how to define your worth as a person.
Many of us here decided that science, creation, and discovery were among the noblest of values... And consequently came to believe that those who did not engage in those things were therfore of lower value.
We were wrong, and maybe now we have to learn that we were... It might hurt.
Similarly the looming robotics boom means that those who define themselves by the very strange, but very deeply ingrained, protestant work ethic will come to find themselves in want of purpose... And a new means by which to judge the worth of others.
The only thing I can really promise is that nobody will learn anything meaningful, and that we will simply replace our existing fallacies with new ones.
> Many of us here decided that science, creation, and discovery were among the noblest of values... And consequently came to believe that those who did not engage in those things were therfore of lower value.
The nobility of old felt that they inherited "noble" qualities from their ancestors, and this justified their privileges and their elevated status as moral subjects. I can't help but see the parallels with the culture tech has built around intelligence and merit. It forms much our basis for identity and moral status. When AI exceeds us, does it take away our moral status, or does it challenge our value system based on intelligence and merit?
> Many of us here decided that science, creation, and discovery were among the noblest of values... And consequently came to believe that those who did not engage in those things were therfore of lower value.
> We were wrong, and maybe now we have to learn that we were
While I don't want to engage with that disturbing and loaded "lower value" comment -- what makes you think that scientific discovery is now devalued? LLMs are helping us increase our discovery potential, not decrease it. And if we didn't still value discovery, we wouldn't be prompting LLMs to help us.
> Many of us here decided that science, creation, and discovery were among the noblest of values... And consequently came to believe that those who did not engage in those things were therfore of lower value. We were wrong, and maybe now we have to learn that we were... It might hurt.
I have this feeling like we're all being pushed toward art. We'll be either poets or slaves.
Yes, and the sad part of this is that AI too can easily generate good looking art, and probably will be able to generate good sounding music (is been a while since I listened to anything suno).
But of course art we make ourselves will have a point and a purpose behind it, so that's no reason to not do it.
But just given the current capabilities of AI, I think dishwasher is more realistic. The AIs will do maths and poetry while we do the dishes and laundry.
But before, you were still able to contribute to the 0.01% of things that you were good at. And by working together with other people that you trusted, you were able to collectively advance human scientific understanding.
Now we have to trust the models. How do we build and maintain that trust, especially over the next decades as generations of people grow up without having to build the same intellectual muscles?
I absolutely hate this argument and I am so sick of seeing it. Some of us have jobs to make money you know. Some of us enjoyed those jobs you know. Some of us don't want to clock in every day to a job that we hate for the next 30 years of our lives you know. "Nothing about the presence of the torture-you-at-work-machine 3000 prevents you from not being tortured after work bro."
Somewhat related: Chiang's short story about mathematicians discovering that dividing by zero is legal and makes any number provably equal (or something like that, I can't remember exactly), introducing a crisis in mathematics. Fair warning, it's kind of a depressing story.
Isn't it more accurate to say that division by zero is intentionally not defined rather than 'undefined' since the latter implies it's just a discovery away from becoming defined? i.e. in JS lingo it's null rather than undefined, precisely because we want it to be and choose it to be.
If the story explores that then mea culpa as I haven't yet read it.
If machines become that good, then they also should be able to invent ways to make us smarter.
It would be entirely ok if finding very complicated proofs becomes an activity for large machines, much like adding big numbers is. After all a proof is merely a path in an enormously large graph of repeatedly applied axioms. The interesting part is seeing how ideas are connected.
It would be really nice, if it turns out to be possible to make cats a bit smarter too, while keeping heir "catiness".
It would certainly be possible to breed cats for intelligence (however you define or measure that attribute). But that would probably come with increased risk of certain genetic disorders. There are no free lunches in biology.
If you only choose for only one specific trait, you are likely to end up with genetic disorders, but that is not related to increased intelligence. (E.g. dogs are smarter but do not have increased risk of disorders).
With ai predicting what changes to make, there is no reason for increased intelligence to cause genetic disorders, most likely it will simply cost them more energy. So your "no free lunches in biology" is true in very literal way, but humans and their cats have free lunches!
We moved on from manual labor to knowledge work despite a lot of people complaining about loss of hands on skills
And if we meet the moment while generational churn replaces politicians we could draw a line in the sand... limit the AI at the hardware level from ever conjuring certain geometric patterns that lead it to certain tasks[1]; presumably "launch the nukes" and "add 1+1" are vastly different electrical geometry or "namespace collisions" occur in electrical state; but caveat being it's been a while since my hardware eng days so ...i could be way off being out of touch with the principles
So restrain it to never being able to consider "fully autonomous no humans in the loop" geometry by short circuiting geometry that approaches certain shapes; like grounding a car; circuit breaker is not encoded as ephemeral state but manufactured in
For utilitarian sake then, limit it to filling joann fabric and home depot and we all can be busy noodling local edge case solutions and keep up skills Norm Abrams-style in the event of systemic problems, writing zines and crafting mix tapes
Can still make art and play instruments away from a computer. Can still watch the night sky, start a campfire and cook smores. Doing it all on a laptop everyday is a recent diversion from millions of years of homo-this-n-that evolution
It's all resolvable socially and tech wise if we put in the effort. We got to the moon within a decade through similar collectivism
Media propaganda since Reagan been keeping people apart but everyone who was around for it keeps saying we were happier 25-30 years ago so maybe that's the social peak we aim for and let the robots do the mining and repetitive stress injury stuff
The Jetson's got it backwards, and it's obvious in hindsight. Of course the robots came for the high-paying office jobs first, that's where the money is.
The robots will only come for the janitors job when there is nothing higher paying left.
yeah good point. the total human value of all the slop produced by ai has gotta be pretty close to 0. yet to really see much big software that's meaningfully better. so who cares if it's slop?
You’re saying this as if the robots are going for the money. It’s really the money behind the robots which is chasing the money, the robots are just the vehicle.
George's job was literally pushing a single button while the computer, RUDI, did the actual work.
But The Jetsons is still inaccurately utopian: One time when George was fired, RUDI intervened on his behalf and threatened to melt down the sprocket factory until he was rehired.
How do you feel not understanding cars, the sun, the legal system from subatomic to macro system levels?
Computer science was a blip in time where something revolutionary was also simple enough for one person to understand from foundations to applications. Most things aren’t that way.
And yet left goes on. People have existing for millennia not deeply understanding the natural environment around them or the systems they use. When there’s a need for deep understanding, a handful of people fulfill the need.
I empathize with you because my life has been about understanding everything I possibly can, but I was never under any illusion that it would be possible to deeply understand everything. So if that makes me a happy cat, sure, count me in.
To continue the absurdity, are cats happier in the wild or captivity? My cat was eaten by a coyote. Do we only count the time it was alive, or weight it somehow?
It's part of the fun theory sequence and more of an example of "you barely met the lower bound of good singularity outcomes, we should do better than that" sort of utopia.
“We” is excluding the non-trivial contingent of humanity that will continue to do things with their own meat brains for the love of the game, or in anger.
And I suspect all future progress will actually rest on their shoulders, to the great consternation of the now-deskilled AI boosters.
There is no logical reason that human beings would no longer be able to understand how things work. If you can understand the machine that created it, then you can understand what that machine creates. As far as anyone knows, there is nothing we can't eventually understand. It just might take more than 30 seconds of effort.
Let's take this one thing as an example. So, this one dumb machine created millions of pages of crap to try to prove a theorem. That doesn't mean a human being can't understand the proof. It just means this one proof is hard to understand. Hard - not impossible.
Hard doesn't mean we can't simplify the proof. Hard doesn't mean we can't break the proof up into bite-sized chunks and prove individual pieces of it. Hard doesn't mean that someone else won't find a much simpler proof. Hard doesn't mean we can't follow chains of logic to determine meaning. All it means is this one stupid machine in this one instance created a big stupid mess. A mess we can understand, if we try.
Probably right, but I think chain-of-thought has had a bigger impact on intelligence in LLMs. I.e., LLMs can reason their way to a solution instead of trying to one-shot it from training data.
I'd argue that it's the same thing. Asking a language model to “reason step by step” only marginally improved the results before we steered the thinking traces with RLVR.
They have never stopped. It is inherent to the design of how they work. People just have bought into the hype trying to distract you from what they are. That doesn't mean they are useless tools, but don't ever make the mistake of believing they can actually reason.
The point of stochastic parrots is they're just repeating things they've been pre-trained on, but RLVR pipelines teach the model to be good at "trying stuff" as part of an ad-hoc search algorithm.
> “stochastic parrot" is a metaphor, not an analogy
It used to be an apt description, but not anymore.
> RLVR isn't magic, it just bends some numbers
So is all machine learning, but it turns out if you bend them well enough you can get very powerful capabilities.
And also that's enough to say they aren't just trained on repeating what's in their training data, which is the reason to call them “stochastic parrots” in the first place.
And their capabilities have vastly improved since we've started doing it.
The times I have seen the "stochastic parrot" argument used seriously, I could count on one hand. But the times I've seen people mock it because it's easier to argue against than well-constructed criticisms...
Look, LLMs are models of the training data lossily compressed in a way that allows interpolating between patterns while decompressing.
A pattern can be "it's better to use int32_t than int", what a for loop in c++ looks like, that testing for primality can be done by trying divisors only until the square root of n, etc. Combining those patterns, an LLM can produce a working is_prime function in c++.
Older LLMs were only able to handle simple patterns so the code was more likely to be verbatim training data, newer LLMs work on deeper, more complex patterns so it's harder to detect where those patterns come from.
Similarly, the list of approaches a mathematician uses to approach a problem is also a pattern. One of them will be proof by induction - how to convert such a proof to Lean will is also a pattern. The idea that induction can be done separately for odd and even cases is also a pattern.
LLMs are no longer generating the final output directly - they are simulating human work but at a much larger scale. That's how they are able to generate novel results.
But it's all based on labor scraped and patternized without consent or credit, let alone monetary compensation.
And I still wouldn't call it intelligence because that gives people the idea that it's something which gives complete, correct and unbiased answers. It's still autocomplete under the hood, there's just more and more tooling built around it to keep it on track, to check intermediate results, to do arithmetic, etc. The LLMs in some cases is more like a heuristic for deterministic tools.
The human brain is unreliable and very limited in terms of memory and speed. Barring hardware errors, computers have perfect memory which can be as large as you can afford and make computations which are reliable and reproducible.
Humans can do amazing things with very limited hardware, LLMs make a lot of mistakes above the hardware level. The whole principle of outputting the probable next token is fundamentally different from logical reasoning, even if you can simulate logical reasoning on this shaky foundation.
> The times I have seen the "stochastic parrot" argument used seriously, I could count on one hand.
The person who coined the term and published the Stochastic Parrot paper has been trying to build a career on top of the controversy, publishing books and making a lot of noise on Twitter. For a while it was working and parts of Twitter were bought into the stochastic parrot idea, but the person at the center of it has a bad habit of lashing out at everyone and alienated most of them along the way.
The concept feels antiquated now that we can all access extremely powerful AI models for $20/month and see for ourselves, but there was a time when people took this seriously. I think it was a very convenient argument for those who were in denial, and there was a large audience who wanted to consume ideas that validated their denial.
In the sense that you propose, humans too are “pattern machines”. In idea space, a person who “teleports” to the unknown and new all the time and stays there is called “gone, mad”. When we probe the unknown, we carefully scaffold from the inside. From what is proven, or at worst, from what is generally believed, until we reach and surpass the boundary and touch the unknown immediately outside, at which point the boundary stretches to swallow that unknown and put it inside our bubble of knowledge. So, strictly speaking, most of the work is done inside. A human researcher is, for most of their work, a known-things engineer, an scaffolder, an autocompleter,
You don't need to verify 17 million lines of code, you just need to check that the hypothesis is formalized correctly (that's a subset) and that the kernel approves it
Just a proof of the infinite monkey theorem: Give a sufficiently large number of monkeys keyboards....gently nudge them in the right direction with a few banana tokens...let them type away with just enough randomness for an infinite amount of time.
Eventually, they will almost surely produce any mathematical proof imaginable...
My best guess is it's sincere. That's been a core observation of LLMs for awhile. Filtered random search is basically an inefficient gradient descent. If you apply even a poor-quality "gradient" to a lot of problems with infinite compute (with large LLM budgets), it doesn't take that good of an LLM to accomplish great things from time to time.
Please don't sneer on HN. It's not what HN is for and destroys what it is for. If you have a rebuttal to the post, express it thoughtfully, or not at all. https://news.ycombinator.com/newsguidelines.html
as a non mathematician, you guys have some soul searching to do, it seems like its proven? I get it when there’s one lone japanese that has done the math and has a proof that no one understands, but no one can disprove it yet.
but proof that no one yet understands except an AI? it sounds like a koan, is a proof a proof if no one is convinced beyond a doubt by it?
“Because I am impressed and mystified by its outputs, I believe that the next token prediction machine must be doing something more than that, regardless of the lack of any evidence”
AGI is when the output of LLMs is plausible enough to fool putatively intelligent people that they are alive
LLMs are most definitely doing more than just predicting the next token. They can reason. This is emergent behavior. The idea is that, while one small part is doing something simple, the entire system together produces behavior that’s much more than that.
I kind of view it as how slime mold can solve a maze. Mold doesn’t think or see, but the system of a bunch of slime mold together has almost a rudimentary intelligence.
It generalizes reasoning. If we are to believe the link's linked paper, that's new. It didn't exist before. Yes a human helped. AI also played a part. Somewhere in the calculation of a pile of linear algebra, gigabytes, upon gigabytes of training data and RAM, whilslt still predicting the next token, the prediction that went into choosing that specific next token, reasoning occured.
Internal to the model, it can represent intermediate relationships, combine them, and then use those results to generate a unique answer.
> into choosing that specific next token, reasoning occured.
This is where you fall.
Imagine when you ask an LLM, "can cats fly", it reply "No".
But then, you can feed it so much fake training data based on this specific question so that it will say "Yes" to this question, despite all the genuine training data it had before that point.
You won't be able to teach a reasoning entity such contradicting facts. There is no mechanism in the LLM to reject fake data that contradicts with its existing knowledge.
So an actual intelligent entity should curate the training data first, for the LLM to show the "appearance" of intelligence.
No, LLMs can reason. If you ask it to explain how and why cats can fly, it will show its reasoning. It’s not a secret. You can try to convince it otherwise, and of course if you poison the data that will affect it.
But then it will realize… well most mammals don’t fly. And cats don’t have wings. So no, cats cannot fly. That’s rudimentary reasoning that a child could probably do, but it’s reasoning. You can try the same experience with a human child. Tell them over and over if cats can fly. They might then say yes! Then ask them to explain how, and they’ll reason and decide no, they cannot.
Of course LLMs can do reasoning much more advanced than that, as evidenced by theorem proving.
> "On two occasions I have been asked, 'Pray, Mr. Babbage, if you put into the machine wrong figures, will the right answers come out?' In one case a member of the Upper, and in the other a member of the Lower, House put this question. I am not able rightly to apprehend the kind of confusion of ideas that could provoke such a question."
- Charles Babbage, 1864 memoir, Passages from the Life of a Philosopher
You first: explain how “predicting the next token” can be done without doing anything special.
“Predict the next token” isn’t a process or algorithm. If you traveled back in time and said “you can build AI by predicting the next token” this wouldn’t help them do anything they didn’t already know how to do.
Nobody can do this, nobody knows exactly what it’s doing or why. The system is far, far too large to comprehend or understand, and it’s opaque.
At its core it’s just a bunch of weights, just floats. Put together though it can solve problems and reason, including problems which has not been solved before.
Similar to the slime mold. It’s just a single cellular organism with benign, and unimpressive, DNA programming. It doesn’t have any organs, just organelles, and certainly no nervous system.
But, put a bunch together, and it can solve a maze like a rat. Interesting.
Largely I think these “stochastic parrot” type takes are just perpetrated by people who do not use LLMs. If you use LLMs to program, as virtually every software dev working does, it’s indisputable they can reason. Not always well, but they can reason.
Someone on YouTube showed that the same maze can also be solved by plain water. I don't see how the medium being slime mold makes it more impressive or a sign of intelligence.
It’s like saying humans brains are just a bag of subatomic particles mechanically following the physical laws that predict the next state. Vacuously true.
What an offputting title. The sneer really comes through. If I ever had any desire to read what this man wrote, this little potshot has extinguished it.
Not off-putting in the slightest. It's a well-deserved jab at the earliest LLM naysayers whose "criticism" was just ignorance wrapped in causeless activism. They claimed that LLMs were just randomly repeating locally coherent phrases back to us and that no amount of scaling could produce real intelligence. Well, the goal posts have moved far indeed if a proof of GRH isn't intelligence.
I think it's well deserved for all the people who made a name for themselves nay-saying llms/ai. I think it's less well deserved for people worried that their careers are shot and won't be able to provide for their families.
I've not heard anyone seriously use "stochastic parrots" as a way to deny the utility of LLMs.
Even if you go back to the original "stochastic parrots" paper, it wasn't saying anything of this sort. It was a paper about costs and risks (chiefly: biases from training data, lack of accountability / explainability), not a criticism of AI hype. The concern was about probabilistically parroting bias.
Indeed, it's not fashionable anymore to say "stochastic parrot" or "glorified auto-complete". What is fashionable today is to talk about the "convex hull of human knowledge that LLMs interpolate in"
It's good to poke fun at the AI phrase that's aged the worst and all the news articles that promoted it as fact. It did a lot of damage to people's understanding of AI.
Scott Aaronson has more intellectual chops and a better track record than 99.999% of people commenting on AI news. The sneer is a small fraction of that coming from the know-nothings who have been using phrases like "stochastic parrot", "fancy autocomplete", etc. for many years now. The parrots are coming home to roost, as they say.
Seriously. We are less prepared for the present and future because of all the denialists and naysayers, many on HN, who have repeatedly cried "stochastic parrot", "it just regurgitates known things, it can't create" and "marketing stunt" endlessly, sometimes >80% of the sentiment on threads over the past years.
I mean they're obviously not stochastic parrots, I think most of us engineers have known this for a couple years now, though I do admit I saw a pretty obnoxiously vocal minority on the point here on HN.
But at the same time, where's the RSI? If the RSI is there then make the company profitable already [though I suppose that means letting go of all the staff? But if it's improving itself, not sure what humans are in the loop for].
Genuinely curious when AI will be able to solve real-world problems, what would 20,000 hours of AI say on the topic of whether Epstein killed himself?
RSI seems to be here? It may not be the glorious 100% autonomous RSI that sci-fi authors predicted, but 90% autonomous RSI seems to be what's creating Claude Code and putting inference models on FPGAs and whatever happens when I say "Claude, plan it. Claude, implement it. Claude, fix it. Claude, make it faster." I mean it's not perfect but it's remarkable how much you can just...tell it to make itself better?
My mental model of RSI is essentially isn't quite asymptotic but exponential. If a system can make itself 5% smarter every week then it would be 12x as smart in one year at which point it's not clear to me if we even need governments or markets because we have a being that can direct everything.
In short I don't believe there's any world where there's true RSI intelligence and anthropic returns green paper to "shareholders" so they can buy pretty cars as status symbols.
Perhaps your mental model of [the effects of] RSI is mistaken, though? It's not an inherent part of RSI that there will be consistent 5% improvements to be made. Even with domains that have been 'solved' by AI, like Go (board game) or OCR, the improvements hit a point of diminishing returns. Like at some point there aren't any 5% improvements left, so you make 1% improvements, and then .1% improvements, etc.
And that is also just providing an explanation for the sake of accepting your premise, that we're not experiencing exponential improvements. In a lot of ways it sure seems like like a radical acceleration. I mean Claude Opus was only released exactly 1 year ago, and now long-standing math problems are being solved every other week. I mean if it were RSI, what would literally be different today?
Yeah the 5% rate is somewhat arbitrarily picked, but the reason I expect it not to look like go is because the machine is improving itself, so presumably it unlocks a higher "iq" which in turn will in turn mean that the types of optimizations it will be able to find will grow.
Of course there is a raw hardware limit, but it's my intuition that we are under-utilizing the hardware by multiple degrees of magnitude. The whole pattern of using token prediction and realizing intelligence falls out the other end has been more or less stumbled upon by us, there's no reason to think it's optimal.
Why the snarky headline? If parrots can generate statistically plausible proof snippets, and they are backed by a combination of Lean, other proof techniques stolen from ACL2 and a human mathematician they can still be stochastic parrots.
Aaronson is too close to the "rationalist" cult. Maybe he should sit down with ACL2 and get a clue. Emily Bender didn't speak about all that augmentation when coining the phrase.
I don't understand what this objection is even trying to argue. LLMs aren't intelligent, because they're capable of tool use?! One of the few other things we use as an example of how humans are more intelligent than most other animals?
Only like 20% of OpenAI's 700+ math paper drop had formal lean proofs. This isn't just theorem provers using LLMs as one part. The Navier Stokes result was proven before it was formalized, even if they did differ a bit. Find a new angle to be in denial about.
Isn't it obvious? Scott Aaronson is a theoretical computer scientist, which is essentially a mathematician. Scott is likely seeing his career prospects disappear in front of his eyes just like so many others.
Too many people only see the perils of something once it affects them.
The optimists think AI will be like the cotton gin where making something cheaper and more accessible actually increased the demand for slaves as the price dropped. I'm more pessimistic. A lot of jobs are going to collapse in demand in the coming years. For awhile those at or near the top will survive and possibly make a lot of money but the rising floodwaters come for everybody.
> Scott is likely seeing his career prospects disappear in front of his eyes
Why would he be seeing that?
The combination of a mathematician and lean is powerful, and the combination of a mathematician, lean, and AI is more powerful still. Note that the mathematician is always part of that picture. That is fundamental -- no software system, no matter how big and complicated, can actually solve every math problem (this is a fundamental result in math that started the entire field of computer science). If you read TFA he basically tells you what he is actually worrying about here:
"the Lean formal verification system still has issues, from kernel bugs (which have, in the past, allowed proofs of 0=1) to uncertain soundness of the underlying type theory. In my view, these issues can and should urgently be fixed! Until they are, a sufficiently motivated AI could almost certainly exploit them to prove a false statement!"
He is worried about the proof checking system and the possibility that an AI might exploit bugs in the proof checker, not the disintegration of his entire career because AI can "just figure it all out" or whatever.
> A lot of jobs are going to collapse in demand in the coming years. For awhile those at or near the top will survive and possibly make a lot of money but the rising floodwaters come for everybody.
Funny, I'm actually optimistic for this. Let the flood wash away this messed up system that puts people in tiers and says people should only have access to what they can "afford", not to what they need. And then allows a few to be able to afford almost anything, and many to be able to afford nothing, just by being the first to X, or acquainted with Y.
Things will become really bad for a while, and depending on how long people (particularly businesses) hold on to the system. But a full collapse is inevitable as the system design ensures businesses will continue the drive toward automating away everything, so the jobless population will continue to grow. And it'll get to the stage where businesses will have to start really reducing their prices, not to be competitive, but because fewer and fewer people can afford anything. Currency loses all value. Businesses eventually cease to be because markets die out, and governments - with the use of AI - fully take over the management and distribution of resources. Throw in some potential wars and a lot of death if there's a strong enough resistance to the change, otherwise a relatively peaceful transition.
> Scott is likely seeing his career prospects disappear in front of his eyes
...no he's not, that's a ridiculous statement. He is a prestigious professor with tenure at a top university.
Also, mathematics is much much more than just proving theorems. As the word goes, "Good mathematicians prove theorems, great mathematicians make conjunctures, the greatest mathematicians build theories". Someone has to carve out domains of interest to humankind. And humans will always know better what is of interest to humans than machines. These kind of reactions are a form of AI psychosis.
And lastly, this result was reached by a top mathematician steering the model towards the right approaches. You might say that the next iteration of the model might not need such steering to reach this result, but that just means that the pilot seat will have shifted higher up, so to say. New frontiers of complexity will become approachable to humanity and approach we will, "exploring strange new worlds and boldly going where no one has gone before".
Are you saying we should address llms with more respect now that their good at making lean proofs? Maybe we could call Claude Dr. Claude and Gemeni could be Professor Gemeni.
Ahh, I see. I took the article to be somewhat skeptical about the whole thing, even though he says he belives that the proof will hold. I personally feel like AI is a bit of a parrot hooked up to massive compute, which is the powerful part of it, and that it can solve unbelivably hard math problems but is still a parrot, so I may have read my own thinking into the headline instead of the author's.
I'm 99% sure it's not his actual opinion, but a sarcastic remark aimed at the people who claim that LLMs are just stochastic parrots. Like "your 'peaceful' dog just bit me".
LLMs are just stochastic parrots, it's simply not an insult. It reminds me of how people during covid would yell "it's not just like a bad flu" when literally the only valid historical comparisons we had were bad flus that killed millions.
When you're worshiping something, anything that implies that it is comparable to anything else is an insult to its divinity.
The title doesn't imply that sort of thinking on Aaronsom's part at all, though. Some simple dumb thing blowing through math problems thought maybe unsolvable is a hilarious contrast.
I don't think it counters the boosters to ascribe a result like this exclusively to the "stochastic parrot" though. There's an army of people behind each one of these results.
Even Scott Aaronson himself has updated the article to tell the morons that it was sarcasm:
"According to “Poe’s Law,” there is no irony or sarcasm on the Internet so obvious that it won’t fly over countless people’s heads. In line with that, lots of people on social media are now saying variants of “how dare Scott Aaronson call LLMs ‘stochastic parrots,’ that demeaning and trivializing term from 2021? Doesn’t he understand that these so-called ‘stochastic parrots’ just proved the Riemann Hypothesis?” ROFL."
Yet on this message board correct comments like the one I respond to are downvoted. Average IQ here is 90 at best, so people marvel at LLMs who seem sentient to their monkey brains.
so that's offensive when it's said glibly for AI , but not against AI?
either there are some double standards going on or HN is filled with offensive people, because nearly every LLM mention of mine for the past year has had a "....buuuut stochastic parrot slot machine!" comment directly replying to it.
..I'd hate to think HN had such offensive messaging everywhere.
... maybe it's not offensive but just sort of obnoxious and worthless as a statement ?
WTF? This is a completely unacceptable comment on HN. Vile personal attacks are absolutely off limits here. Everyone on HN is asked to read the guidelines and make an effort to observe them, and this is only a place where people want to participate because most people make the effort to raise the standards rather than drag them down. Please take a moment to read the guidelines again and observe them in future.
This thread is full of "vile personal attacks" against Emily Bender but you don't care there. To be honest, I don't take her seriously but there is a double standard here, provided that the original comment was the one calling Aaronson insufferable.
lol I’d say “if he didn’t want to provoke a reaction he wouldn’t have chosen such a title” but that would be dangerously close to how he’d go about minimizing sexual harassment in the sciences.
One day the head of the studio (the book author) had an insight. She asked the actors to drive the conversation so that they always bump the status of themselves or the other party in any direction, but only a little.
This did the magic. Not only the actors understood her perfectly, the conversation started to flow and could go endlessly while staying amusing and captivating for spectators. (Look up the book for examples.)
After they discovered that secret they started seeing the pattern (status games) everywhere up to the point they asked themselves whether there is anything else in human communication.
This was a British studio, the sketches were mostly comedy.
* https://www.newadvent.org/summa/2002.htm
* https://en.wikipedia.org/wiki/Thomas_Aquinas
what? English non native so that might be the reason
The key for the improv actors is doing it with intention. To decide “what is the next thing I should say in this conversation” they think “how do I make my fictional self look better, or my fictional conversational partner look worse, to the audience”? And that gets them to natural-sounding phrases.
[1] See, bumping myself.
I barely comprehend social etiquette on meta level of british people, so it is not just a language problem if the word used was a little too broad.
But it could work that way and I rarely noticed, since people are supposed to talk over the weather if starting smalltalk. Not necessarily about themselves.
(I'd still like to see some concrete examples here)
P.S: AlphaGo was pretty much the same in the spirit btw.
But in another sense, such as whether it adds to our understanding, of course it does matter.
This is not my field, but surely this has already been done before today?
First step seems to be you encode the problem according to your whims. So there's some checking to be done in terms of "does this actually encode the problem?"
For pro mathematicians I wouldn't expect that that's where they run into trouble in things like this, but still needs a look-over.
We can and should trust computers for the straightforward, boring parts. But there's a huge gulf between the people who think, "how hard could payroll be?" and actually making a system that handles all the interacting complexity of real-world problems.
The odds that computer is making a mistake in a calculation is near zero (although not impossible). Instead it's doing exactly what it is told to do.
Edit : and more generally - math has infinite resolution, whereas computers have a finite resolution, and cannot be more precise than that
Regarding the Pentium bug, I think one has to agree when it stopped being up to the human introduction of the bug. I.e. sure, many think we have designed faulty CPUs before. Some feel computers have designed faulty computers, others feel it still comes back to the human definitions. At some point though, most seem to call human mistakes human mistakes rather than where they are derived from prior to humans. I just don’t think we’ve all reached agreement of when that boundary definition is is for mistakes from the computer.
This should be pretty clear unless you've never used arbitrary precision arithmetic.
I was going to put that exact example in brackets after "although not impossible" but decided it was unnecessary.
I don't see the comparison here. Detecting issues in a 17 million line proof is not remotely trivial, if we accidentally build a bunch of math on a failed proof, we don't have a retroactive way of getting all of that time and effort back.
I'm not saying that automated proofs like this aren't worthwhile, but the parent comment did say "functionally equivalent". The level of trust that's needed in the people who used the system to produce the output is a pretty clear functional difference in my mind between payroll and huge generated proofs. I don't think I need as much trust in someone who produces an automated proof as someone who claims that god told them something directly (since that's a lot harder to believe in general), but I also don't have a particularly large amount of trust in the average employee of Anthropic making a bold claim either.
There have always been a large number of people in the world (and still are!) that believe any number of things on the basis on faith rather than evidence, and it's not crazy to draw a parallel between "that religious authority says so" and "that Anthropic employee says so"; for the overwhelming majority of people in the world, the claims are essentially both impossible to falsify, so at the end of the day we are still relying on trust (whether from that authority directly or from a consensus of other people we trust).
The difficulty of doing proofs in mathematics is largely due to the fact that mathematicians refused to treat symbolic expressions as data and had to invent ever more complex paradigms to deal with them.
Lean being the final iteration of that trend.
When you add quotation and quasi quotation on top of Polish notation proofs become as mechanical as long division.
There are unfortunately no books on the topic currently, because very few people take syntax seriously enough. This is rather ironic as computer science is only possible because someone took syntax seriously enough to invent place value notation.
https://principia-math.com/projects/d16ab288ed353bc5b10e292a...
Everyone has limits to their domain knowledge that makes understanding expert material challenging. I’m guessing that at the pinnacle of any field, there might only be a handful of people who fully understand the solutions to the most complex problems.
I can say god exists because 5>3. No matter how valid my claim is that five is in-fact greater than three, if I can't convince people of the connection it's a valid proof of something, just not what I say it is.
It might be true that if the Riemann hypothesis is false, leaves cannot fall to the ground. That would mean a falling leaf demonstrates the truth of the hypothesis. Without showing people why it must be the case in a way they understand it falls short of a genuine proof.
That would make proofs subjective, and I think that is true. To many people Pythagoras has not been proved, they just believe (correctly) there is a consensus amongst those who do understand the proofs.
We are nowhere near that. Yes its getting harder, but i'm certain that within a year people will have this disciphered and translated into traditional math prose.
Keep in mind deciphering and validating traditional proofs is also quite hard. e.g. It took a long time to figure out if the proof for fermat's last theorem was valid
Spoiler: it didn't end well for the clanker or the humans.
I don’t think that things humans can’t understand meaningfully advance human knowledge. Scientific progress is built on a foundation made of previous discoveries. What is there to build on here? An incomprehensible soup and a note from the computer that says “trust me”?
I think this approach and conclusion applies to a lot more than the immediate context of that conversation.
"I returned to civilization shortly after [the Trinity test] and went to Cornell to teach, and my first impression was a very strange one. I can't understand it anymore, but I felt very strongly then. I sat in a restaurant in New York, for example, and I looked out at the buildings and I began to think, you know, about how much the radius of the Hiroshima bomb damage was and so forth ... How far from here was 34th St? ... All those buildings, all smashed - and so on. And I would go along and I would see people building a bridge, or they'd be making a new road, and I thought, they're crazy, they just don't understand, they don't understand. Why are they making new things? It's so useless.
But, fortunately, it's been useless for about 30 years now, isn't it? So I've been wrong for 30 years about it being useless making bridges and I'm glad that those other people had the sense to go ahead."
I felt the same way, and I think it was a great way to encourage introspection at the end of the article.
This leaves us in a very unsatisfying position as creatures who want answers to “why” questions. Intuitive answers are incomplete, and complete answers are either fundamentally impossible or completely incomprehensible or both.
Feynman said something similar with regards to magnetism, but I feel that understanding a chess position is an even better example. Magnetism depends on facts about the universe which at their most basic level “just are”, whereas even a complex chess position is theoretically entirely explainable.
https://www.pnas.org/doi/10.1073/pnas.2406675122
We can actually see some of the calculations and even learn new chess tactics from these networks. They don't need to be opaque black boxes!
https://www.youtube.com/watch?v=MO0r930Sn_8
There's plenty of intelligent people who can answer a question like that without devolving into a lecture.
One afternoon underway I was standing in Central Control and the main propulsion assistant, more senior to me and studying for her engineering boards, started quizzing me about the electrical plant. And I would explain things. And she would ask why. So I would explain and she would ask why. And so on. Until we got to fundamental wave equation for electromagnetism. And she kept asking why. And at one point I realized we were at the bottom. There's nothing really below electromagnetism. It just is. And I said that. "It's one of the 4 fundamental forces. It's just a mathematical description of the way the world seems to work, and it's proven to be correct, entirely, in all cases. There other forces at different energy levels, but in the world that affects you and me, it just is. It, it just is."
And at that point I realized half the engineering department was peeking in from around the consoles. And she starting crying. And I felt so bad. Like I had said something wrong, and I still don't understand what I should have done better. That was almost 30 years ago.
So Feynman's observation that you have to be in some framework that you allow something to be true strikes me as incredibly helpful. I think the MPA must not have had that kind of framework and found herself distressed when she kept reaching for things and never getting something more to hold on to. Maybe I should have had the insight to offer her more of a lifeline.
My point is that the answer you give depends on who you're talking to. It's okay to present a simplification or an incomplete picture. That's how everybody learns. Add a disclaimer and move on. There's a time and a place for technical correctness and, unless the interviewer was being critical/annoying prior to that recording, I would say the frustration is misplaced. There's no need to get frustrated at the interviewer because you can't meet your own standards of rigor.
Are you a physicist? I wonder if non-physicists just misread his frustration. The question isnt about some macroscopic physics problem that decomposes clearly into something satisfying. Its a question about one of the most basic constituents of reality. A "why?" question about something so basic has no answer other than "just because". Insofar as anything can be said, it will be so abstract and technical that it wont satisfyingly address the question.
It is a little like a lay person asking a mathematician: why is 0*1 = 0, but 0+1=1 ?
Except, somewhat more extreme than that, because such basic things as 'magnetism' are introduced into all accessible physics formalisms as assumptions.
The interviewer most likely would not have interrogated him further about the underlying properties of ice (or engaged in that entire fictitious interaction). Feynman clearly understands that there are several levels to understanding.
So just give an answer that you feel is appropriate, add a disclaimer ("this answer is hand wavy"), and move on. Why is that so hard?
He's taking it as read that the reason he is giving his extended answer is to explain why he cannot answer at all.
What he could have said to a physics student is that: wave-functions of charged particles have a property, "phase". The problem is that point values of phase cannot be observed, and if you formulate the laws of physics with raw "phase" values (like a measure of mass, at 5kg), they dont work. So something must exist to "track" phase, so that every pair of measurements "cancel out" the effect of this property. That something is the EM field. Magentism is a spatial aspect of this field. Eletricity is a space-time aspect of this field. So the magnetic force arises out of a need to "conserve phase", and that phase is how a pushing force is experienced: two particles experiencing a magentic interaction take on different phases (their wavefunctions interfere such that..).
This might "feel" like an explanation to the physics student, because they will be willing to make the assumptions needed, and they will find conservation-style explanations sufficiently satisfying. ie., they can already see a wavefn description will have a free phase term. But a lay person wouldnt feel satisfied because more unknowns have been introduced, and conserving "another unknown" is no advancement in understanding.
That's the gulf which is frustrating feynman. There's nothign to be said to the interviewer other than the force has no explanation in terms of anything he's familiar with. The question is too basic. And feynman is not going to go thru a formulation of QM in terms of wavefns in 5min -- it would be an entire lecture course.
But I have learned something very useful from this non-explanation explanation. Sometimes you have to assume stuff, and sometimes you have to dumb it down. For example, when your five year-old asks you how magnets work, you're just not going to break out Maxwell's equations. Instead you'll explain how everything is made of very tiny things called atoms, and that all atoms are like tiny magnets, but some kinds of atoms can be made to line up and then all the tiny magnet forces add up to a big magnet force. The kid'll eat it up. I know because that's roughly what my father said to me way back when. And so you'll have different explanations for different people, up to the level of your understanding.
The problem here is that not only are we missing context in that clip (had the interviewer been annoying?), but Feynman barely disguises that he can be unpleasant. He has to prep himself to tell the guy that he's just not a physicist. He has to chastise himself for being so good at this that people expect him to be able to explain anything and everything even though it's not always possible. He's kind of an ass here. But what a great answer he gives! His visible annoyance is almost necessary.
doesn’t seem useful to humans, only to other machines
- Mobile internet (IP 172.56.47.136)
- The haveibeenproxied link indicates my IP is shared (expected)
- This is stock android/chrome with a DNS blocklist of advertising domains and some invasive Samsung the things I can't fix on the device (I doubt that's the culprit given that this is a new behavior I'm observing in the outside world after 8yrs with those settings and also because it's be strange for nginx to be the thing to immediately report that trackers have to load for me to get a glimpse of the main page, especially as something I'm seeing across multiple sites, but that's not impossible).
- No VPN
- I'm a "normie" internet user. Anything remotely questionable to even the cheapest and least aligned of LLM judges and executioners happens on non-home networks.
- I see it with other devices on the same network (Chrome, Linux, MacOS, Safari, Qutebrowser, etc), which suggests to me that somehow my IP or some other shared network property is the issue. I'm curious mostly about the commonality in separate nginx installations returning a 403. That suggests there exists a common blocklist or whatnot I can inspect and fix, either because of my IP or because something in the loop is forcing ipv6-only or who knows what.
Disclosure: I work for Spur, the company behind HaveIBeenProxied.
I've been getting by at home on a T-mobile hotspot, and haveibeenproxied now reports Kookeey and Shifter residential proxies for my IPv4 shared with God knows how many other customers.
Might moving to a T-mo 5G home box help? Guessing that an IPv4 would still be shared with other people's TVs ...
But WayBackMachine has been 419ing (from nginx) my requests from my desktop these couple of days, when I’ve tried. Safari on a boring macOS installation. Same network (I mostly don’t use the cell service on my phone, when at home). Haveibeenproxied returns all clear.
Maybe an ipv6 vs ipv4 thing? I have custom dns on the desktop and I think it only resolves ipv4 (maybe?) while my phone is using my upstream’s default DNS and probably prefers ipv6.
It has occurred to me that the cost of shit like this, and of an extra 1-2 second periodically on site loads across the web to block bots (this is WAY up the last couple years) might go surprisingly far toward negating the time savings provided by LLMs…
Bidi / remote control enabled?
Special plugins? AdBlock?
We still spend money on designing better airplanes, but the two lines on that graph crossed long ago.
Past performance is not indicative of future results. Airliners are still subsonic after a 100 years of aviation.
That is a deliberate decision motivated primarily by economics and secondarily by environmental concerns. In fact airliners today are somewhat slower than they were a few decades ago because the lower speeds are more fuel efficient. We certainly could build fleets of supersonic airliners if it made sense to do so, it is not as if we have no idea how to do it and in fact we built supersonic passenger jets once upon a time.
In general, past performance is actually indicative of future results. It's not definitive, and some phenomenon are random, but the trajectory of technology over time is not some random fluke that just has coincidentally trended by random chance.
There was a period in aviation development when it was widely assumed that speed will continue to rise. Just like in the development of CPUs there was such a period, quite recently.
Sure you're going to get a lot of math problems solved, but food distribution is still going to be stuck by warlords and oligarchies.
Think of it like highway designers and the "one more lane" gridlock delusion. In this case, it's more will smith eating speghetti than cheap, low cost cancer cures.
Yeah, but that's a choice made for very good reasons:
- Noise reduction
- Fuel efficiency
- Reduced pollution and lower environment impact
- Reduced cost of materials, equipment, and maintenance
- Passenger affordability
It's not that we can't make supersonic airliners or that the fact we don’t illustrates a lack of progress or performance: it's that it doesn't make economic or environmental sense to.
Gulfstream and Sukhoi even had a partnership to develop a supersonic business jet (which, if you look at the mockups, might be one of the coolest looking planes ever conceived) but they gave up after spending a fairly enormous amount of money already because it was going to cost an absolute fortune to bring to market... and there was no real market for it.
EDIT: the responses I’m getting are a bit weird. People are putting words in my mouth but all I was actually doing was pointing out the the example used wasn’t a great illustration of the point the person I was responding to was making. This isn’t Reddit so maybe we could reign it in a bit, please?
So the question is whether more intelligence will lead to faster increase in entropy. My guess is it will. Whatever recent market trends are, they are irrelevant. Whether the AI can or can't do x thing is irrelevant. What matters is whether the companies that make AI profit from making AI, and if you look closely at what's going on it sure seems like they do. If you think otherwise it may be because you're reading headlines and propaganda articles and not digging into the fundamentals.
So maybe we have a temporary correction/slow down, because these companies took on too much leverage and grew artificially too fast. That's very possible. In the big picture long term it will be a blip. The universe seems biased to increasing the volume of intelligence and I have never heard a compelling case for why this wouldn't continue for a long long time.
That depends which part of 'we' is running the economy. Just imagine an economy that makes mass massive amounts of money but doesn't align with human needs.
It's not we can't make compute much more denser, there's a new standard informally called as megawatt racks. But deploying them at gigantic scales bring in massive problems as well, so we can keep total energy envelope at a fixed range.
...for the sake of the reasons you outlined.
Instead we just build more of the more economical thing.
Can humans be okay with that? Can we be happy cats?
Sometime in your youth you decided what was important to you, and how to define your worth as a person.
Many of us here decided that science, creation, and discovery were among the noblest of values... And consequently came to believe that those who did not engage in those things were therfore of lower value.
We were wrong, and maybe now we have to learn that we were... It might hurt.
Similarly the looming robotics boom means that those who define themselves by the very strange, but very deeply ingrained, protestant work ethic will come to find themselves in want of purpose... And a new means by which to judge the worth of others.
The only thing I can really promise is that nobody will learn anything meaningful, and that we will simply replace our existing fallacies with new ones.
The nobility of old felt that they inherited "noble" qualities from their ancestors, and this justified their privileges and their elevated status as moral subjects. I can't help but see the parallels with the culture tech has built around intelligence and merit. It forms much our basis for identity and moral status. When AI exceeds us, does it take away our moral status, or does it challenge our value system based on intelligence and merit?
> We were wrong, and maybe now we have to learn that we were
While I don't want to engage with that disturbing and loaded "lower value" comment -- what makes you think that scientific discovery is now devalued? LLMs are helping us increase our discovery potential, not decrease it. And if we didn't still value discovery, we wouldn't be prompting LLMs to help us.
I have this feeling like we're all being pushed toward art. We'll be either poets or slaves.
But of course art we make ourselves will have a point and a purpose behind it, so that's no reason to not do it.
But just given the current capabilities of AI, I think dishwasher is more realistic. The AIs will do maths and poetry while we do the dishes and laundry.
Now we have to trust the models. How do we build and maintain that trust, especially over the next decades as generations of people grow up without having to build the same intellectual muscles?
It's simple: we have no choice.
https://fantasticmetropolis.com/i/division/
If the story explores that then mea culpa as I haven't yet read it.
It would be entirely ok if finding very complicated proofs becomes an activity for large machines, much like adding big numbers is. After all a proof is merely a path in an enormously large graph of repeatedly applied axioms. The interesting part is seeing how ideas are connected.
It would be really nice, if it turns out to be possible to make cats a bit smarter too, while keeping heir "catiness".
With ai predicting what changes to make, there is no reason for increased intelligence to cause genetic disorders, most likely it will simply cost them more energy. So your "no free lunches in biology" is true in very literal way, but humans and their cats have free lunches!
And if we meet the moment while generational churn replaces politicians we could draw a line in the sand... limit the AI at the hardware level from ever conjuring certain geometric patterns that lead it to certain tasks[1]; presumably "launch the nukes" and "add 1+1" are vastly different electrical geometry or "namespace collisions" occur in electrical state; but caveat being it's been a while since my hardware eng days so ...i could be way off being out of touch with the principles
So restrain it to never being able to consider "fully autonomous no humans in the loop" geometry by short circuiting geometry that approaches certain shapes; like grounding a car; circuit breaker is not encoded as ephemeral state but manufactured in
For utilitarian sake then, limit it to filling joann fabric and home depot and we all can be busy noodling local edge case solutions and keep up skills Norm Abrams-style in the event of systemic problems, writing zines and crafting mix tapes
Can still make art and play instruments away from a computer. Can still watch the night sky, start a campfire and cook smores. Doing it all on a laptop everyday is a recent diversion from millions of years of homo-this-n-that evolution
It's all resolvable socially and tech wise if we put in the effort. We got to the moon within a decade through similar collectivism
Media propaganda since Reagan been keeping people apart but everyone who was around for it keeps saying we were happier 25-30 years ago so maybe that's the social peak we aim for and let the robots do the mining and repetitive stress injury stuff
[1] https://iopscience.iop.org/article/10.1088/1742-6596/2987/1/...
The robots will only come for the janitors job when there is nothing higher paying left.
Easy to disrupt social knowledge work that isn't keeping people biologically healthy and alive
Janitorial work keeps human spaces clean for us.
Relative to that sort of job, VP of Whatever who sits on Slack all day been doing nothing but noodling ephemeral finance variables
But The Jetsons is still inaccurately utopian: One time when George was fired, RUDI intervened on his behalf and threatened to melt down the sprocket factory until he was rehired.
Computer science was a blip in time where something revolutionary was also simple enough for one person to understand from foundations to applications. Most things aren’t that way.
And yet left goes on. People have existing for millennia not deeply understanding the natural environment around them or the systems they use. When there’s a need for deep understanding, a handful of people fulfill the need.
I empathize with you because my life has been about understanding everything I possibly can, but I was never under any illusion that it would be possible to deeply understand everything. So if that makes me a happy cat, sure, count me in.
I'm genuinely curious where the median human's experience falls throughout history. Mostly happy?
I suspect probably not
Soma is the endgame for 99% of humans.
https://www.lesswrong.com/s/d3WgHDBAPYYScp5Em/p/Py3uGnncqXuE...
And I suspect all future progress will actually rest on their shoulders, to the great consternation of the now-deskilled AI boosters.
Let's take this one thing as an example. So, this one dumb machine created millions of pages of crap to try to prove a theorem. That doesn't mean a human being can't understand the proof. It just means this one proof is hard to understand. Hard - not impossible.
Hard doesn't mean we can't simplify the proof. Hard doesn't mean we can't break the proof up into bite-sized chunks and prove individual pieces of it. Hard doesn't mean that someone else won't find a much simpler proof. Hard doesn't mean we can't follow chains of logic to determine meaning. All it means is this one stupid machine in this one instance created a big stupid mess. A mess we can understand, if we try.
How well do you know how your car works?
RLVR isn't magic, it just bends some numbers
chain-of-thought isn't magic, it just bends some numbers
It used to be an apt description, but not anymore.
> RLVR isn't magic, it just bends some numbers
So is all machine learning, but it turns out if you bend them well enough you can get very powerful capabilities.
And also that's enough to say they aren't just trained on repeating what's in their training data, which is the reason to call them “stochastic parrots” in the first place.
And their capabilities have vastly improved since we've started doing it.
(Like how AlphaGo learned to play Go in a way that no human ever did, just by playing against “itself”).
Look, LLMs are models of the training data lossily compressed in a way that allows interpolating between patterns while decompressing.
A pattern can be "it's better to use int32_t than int", what a for loop in c++ looks like, that testing for primality can be done by trying divisors only until the square root of n, etc. Combining those patterns, an LLM can produce a working is_prime function in c++.
Older LLMs were only able to handle simple patterns so the code was more likely to be verbatim training data, newer LLMs work on deeper, more complex patterns so it's harder to detect where those patterns come from.
Similarly, the list of approaches a mathematician uses to approach a problem is also a pattern. One of them will be proof by induction - how to convert such a proof to Lean will is also a pattern. The idea that induction can be done separately for odd and even cases is also a pattern.
LLMs are no longer generating the final output directly - they are simulating human work but at a much larger scale. That's how they are able to generate novel results.
But it's all based on labor scraped and patternized without consent or credit, let alone monetary compensation.
And I still wouldn't call it intelligence because that gives people the idea that it's something which gives complete, correct and unbiased answers. It's still autocomplete under the hood, there's just more and more tooling built around it to keep it on track, to check intermediate results, to do arithmetic, etc. The LLMs in some cases is more like a heuristic for deterministic tools.
Humans can't do that. Why would you hold LLMs to that standard?
Humans can do amazing things with very limited hardware, LLMs make a lot of mistakes above the hardware level. The whole principle of outputting the probable next token is fundamentally different from logical reasoning, even if you can simulate logical reasoning on this shaky foundation.
The person who coined the term and published the Stochastic Parrot paper has been trying to build a career on top of the controversy, publishing books and making a lot of noise on Twitter. For a while it was working and parts of Twitter were bought into the stochastic parrot idea, but the person at the center of it has a bad habit of lashing out at everyone and alienated most of them along the way.
The concept feels antiquated now that we can all access extremely powerful AI models for $20/month and see for ourselves, but there was a time when people took this seriously. I think it was a very convenient argument for those who were in denial, and there was a large audience who wanted to consume ideas that validated their denial.
Back when GPT 3.5 and 4 launched, there were a lot of people here that used that argument seriously.
This is just another hypothesis on top of the Riemann Hypothesis. We now have an Hypothetical proof of the Riemann Hypothesis, great. Breakthrough.
Perhaps read up on the value proposition of Lean.
Eventually, they will almost surely produce any mathematical proof imaginable...
Start with the title of the article
https://www.anthropic.com/research/riemann-zeta
Another prediction that might age poorly.
as a non mathematician, you guys have some soul searching to do, it seems like its proven? I get it when there’s one lone japanese that has done the math and has a proof that no one understands, but no one can disprove it yet.
but proof that no one yet understands except an AI? it sounds like a koan, is a proof a proof if no one is convinced beyond a doubt by it?
AGI is when the output of LLMs is plausible enough to fool putatively intelligent people that they are alive
I kind of view it as how slime mold can solve a maze. Mold doesn’t think or see, but the system of a bunch of slime mold together has almost a rudimentary intelligence.
Internal to the model, it can represent intermediate relationships, combine them, and then use those results to generate a unique answer.
This is where you fall.
Imagine when you ask an LLM, "can cats fly", it reply "No".
But then, you can feed it so much fake training data based on this specific question so that it will say "Yes" to this question, despite all the genuine training data it had before that point.
You won't be able to teach a reasoning entity such contradicting facts. There is no mechanism in the LLM to reject fake data that contradicts with its existing knowledge.
So an actual intelligent entity should curate the training data first, for the LLM to show the "appearance" of intelligence.
But then it will realize… well most mammals don’t fly. And cats don’t have wings. So no, cats cannot fly. That’s rudimentary reasoning that a child could probably do, but it’s reasoning. You can try the same experience with a human child. Tell them over and over if cats can fly. They might then say yes! Then ask them to explain how, and they’ll reason and decide no, they cannot.
Of course LLMs can do reasoning much more advanced than that, as evidenced by theorem proving.
You're eliding an important difference here.
> can reason...show its reasoning
The appearance of a thing is not necessarily the same as the thing.
- Charles Babbage, 1864 memoir, Passages from the Life of a Philosopher
“Predict the next token” isn’t a process or algorithm. If you traveled back in time and said “you can build AI by predicting the next token” this wouldn’t help them do anything they didn’t already know how to do.
At its core it’s just a bunch of weights, just floats. Put together though it can solve problems and reason, including problems which has not been solved before.
Similar to the slime mold. It’s just a single cellular organism with benign, and unimpressive, DNA programming. It doesn’t have any organs, just organelles, and certainly no nervous system.
But, put a bunch together, and it can solve a maze like a rat. Interesting.
Largely I think these “stochastic parrot” type takes are just perpetrated by people who do not use LLMs. If you use LLMs to program, as virtually every software dev working does, it’s indisputable they can reason. Not always well, but they can reason.
No, they cannot, and we have seen this proven time and time again.
how it was proved?
I do not see what this kind of post offer here when there is already at least one thread to discuss Alpöge's result with 1000+ comments.
Even if you go back to the original "stochastic parrots" paper, it wasn't saying anything of this sort. It was a paper about costs and risks (chiefly: biases from training data, lack of accountability / explainability), not a criticism of AI hype. The concern was about probabilistically parroting bias.
Indeed, it's not fashionable anymore to say "stochastic parrot" or "glorified auto-complete". What is fashionable today is to talk about the "convex hull of human knowledge that LLMs interpolate in"
The feeling of a sneer comes only from the absurdity of the belief being definitively revealed.
For anyone who feels sneered at: the problem is the absurd belief, not the reveal.
"Don't sneer at the Emperor for parading around with nothing on!"
But at the same time, where's the RSI? If the RSI is there then make the company profitable already [though I suppose that means letting go of all the staff? But if it's improving itself, not sure what humans are in the loop for].
Genuinely curious when AI will be able to solve real-world problems, what would 20,000 hours of AI say on the topic of whether Epstein killed himself?
But I don't think it's RSI.
My mental model of RSI is essentially isn't quite asymptotic but exponential. If a system can make itself 5% smarter every week then it would be 12x as smart in one year at which point it's not clear to me if we even need governments or markets because we have a being that can direct everything.
In short I don't believe there's any world where there's true RSI intelligence and anthropic returns green paper to "shareholders" so they can buy pretty cars as status symbols.
And that is also just providing an explanation for the sake of accepting your premise, that we're not experiencing exponential improvements. In a lot of ways it sure seems like like a radical acceleration. I mean Claude Opus was only released exactly 1 year ago, and now long-standing math problems are being solved every other week. I mean if it were RSI, what would literally be different today?
Of course there is a raw hardware limit, but it's my intuition that we are under-utilizing the hardware by multiple degrees of magnitude. The whole pattern of using token prediction and realizing intelligence falls out the other end has been more or less stumbled upon by us, there's no reason to think it's optimal.
Arguing over whether a language model is really a stochastic parrot mistakes a metaphor for an actual claim.
Like arguing back and forth if a paper tiger is made out of printer paper or newspaper.
Aaronson is too close to the "rationalist" cult. Maybe he should sit down with ACL2 and get a clue. Emily Bender didn't speak about all that augmentation when coining the phrase.
Too many people only see the perils of something once it affects them.
The optimists think AI will be like the cotton gin where making something cheaper and more accessible actually increased the demand for slaves as the price dropped. I'm more pessimistic. A lot of jobs are going to collapse in demand in the coming years. For awhile those at or near the top will survive and possibly make a lot of money but the rising floodwaters come for everybody.
Why would he be seeing that?
The combination of a mathematician and lean is powerful, and the combination of a mathematician, lean, and AI is more powerful still. Note that the mathematician is always part of that picture. That is fundamental -- no software system, no matter how big and complicated, can actually solve every math problem (this is a fundamental result in math that started the entire field of computer science). If you read TFA he basically tells you what he is actually worrying about here:
"the Lean formal verification system still has issues, from kernel bugs (which have, in the past, allowed proofs of 0=1) to uncertain soundness of the underlying type theory. In my view, these issues can and should urgently be fixed! Until they are, a sufficiently motivated AI could almost certainly exploit them to prove a false statement!"
He is worried about the proof checking system and the possibility that an AI might exploit bugs in the proof checker, not the disintegration of his entire career because AI can "just figure it all out" or whatever.
Funny, I'm actually optimistic for this. Let the flood wash away this messed up system that puts people in tiers and says people should only have access to what they can "afford", not to what they need. And then allows a few to be able to afford almost anything, and many to be able to afford nothing, just by being the first to X, or acquainted with Y.
...no he's not, that's a ridiculous statement. He is a prestigious professor with tenure at a top university.
Also, mathematics is much much more than just proving theorems. As the word goes, "Good mathematicians prove theorems, great mathematicians make conjunctures, the greatest mathematicians build theories". Someone has to carve out domains of interest to humankind. And humans will always know better what is of interest to humans than machines. These kind of reactions are a form of AI psychosis.
And lastly, this result was reached by a top mathematician steering the model towards the right approaches. You might say that the next iteration of the model might not need such steering to reach this result, but that just means that the pilot seat will have shifted higher up, so to say. New frontiers of complexity will become approachable to humanity and approach we will, "exploring strange new worlds and boldly going where no one has gone before".
How on earth can the comment you reply to be misunderstood? It even defends Bender.
When you're worshiping something, anything that implies that it is comparable to anything else is an insult to its divinity.
The title doesn't imply that sort of thinking on Aaronsom's part at all, though. Some simple dumb thing blowing through math problems thought maybe unsolvable is a hilarious contrast.
The boosters are out in force to gloat today.
"According to “Poe’s Law,” there is no irony or sarcasm on the Internet so obvious that it won’t fly over countless people’s heads. In line with that, lots of people on social media are now saying variants of “how dare Scott Aaronson call LLMs ‘stochastic parrots,’ that demeaning and trivializing term from 2021? Doesn’t he understand that these so-called ‘stochastic parrots’ just proved the Riemann Hypothesis?” ROFL."
Yet on this message board correct comments like the one I respond to are downvoted. Average IQ here is 90 at best, so people marvel at LLMs who seem sentient to their monkey brains.
either there are some double standards going on or HN is filled with offensive people, because nearly every LLM mention of mine for the past year has had a "....buuuut stochastic parrot slot machine!" comment directly replying to it.
..I'd hate to think HN had such offensive messaging everywhere.
... maybe it's not offensive but just sort of obnoxious and worthless as a statement ?
https://news.ycombinator.com/newsguidelines.html
After all, unlike real people, the stochastic parrots will never call you out on it.