A misalignment of AI in mathematics (mathandai.org)

Dr. Tao said the same thing. Somehow, this letter came through. He wants to conduct Math competitions where participants who don’t have formal credentials can contribute to mathematical research through AI.

Title: Terence Tao - SAIR Competitions and the Future of Experimental Mathematics

https://www.youtube.com/watch?v=rB9YOi3lb7w

and this:

Daniel Litt - Working with LLMs to do high quality math

https://www.youtube.com/watch?v=0wL8NlhxXcU

Lee Sedol said in an interview that "losing to AI, in a sense, meant my entire world was collapsing. ... I could no longer enjoy the game. So I retired", and I think there will be folks in the mathematical community who would feel the same when the solutions pages to hard problems are suddenly available.

But on the other hand, people learned a lot from chess engines. After decades of chess computers beating humans, there was still a renewed interest in watching Leela beat Stockfish, with many people trying to understand the strategy Leela used.

If your happiness comes from grinding on a problem and making progress, the prospect of having to dig through a corpus of AI-generated proofs might be hard to swallow. But if you're willing to do that, you will still find beautiful things that only so many people can truly appreciate.

It'll basically become slop fatigue if OpenAI starts dumping out proofs faster than the community can keep up, and some turn out to be wrong, never formalize it, don't stay to support it, etc.

Mochizuki was still one human and it required legions of other humans to unpack and untangle to confirm that it didn't lead to anywhere in particular.

AI is now capable of constructions so complex that no human or human team can unpack. And its ability to increase that complexity is growing while our human ability is stagnant.

meta-AI analysis cannot help. We (software professionals who use AI regularly) already know that if you run into a situation where a Fable/Astra-generated analysis reaches the limits of our comprehension/complexity due to their subjectivity, throwing more AI at the problem doesn't always converge.

There are many reasons to feel optimistic about AI, and ultimately its general ability to help science and mathematics.

I see no reason to feel optimistic about the future of mathematics and AI based on the current path of frontier labs, unless the misalignment Tao is writing about can be reconciled.

Imagine a world where these most complex mathematical problems are not accessible to a few hundred people, but a few hundred thousands people. ...Those original few hundred gifted mathematicians would have an even more prominent role, and their names and achievements would be known by orders of magnitude more people that they are now.

I don't like nuance here. I think progress is really measured by what humans are able to do and understand, not machines. It is significant if we find problems we struggle to solve. That tells us something. What does it take for humans to solve these problems is related.

The best analogy I can give is if you wanted to climb Mt. Everest you might ask someone for guidance. Would it be better to ask someone who has climbed Mt. Everest or someone who took a helicopter ride up near the top and then went to the peak? This is like the AI versus human gap to me. The helicopter is like using AI to generate a proof. The person who actually climbed Mt. Everest has firsthand knowledge of the experience. Same thing for a difficult proof. The struggle people have is actually valuable here. Likewise, we know people are actually capable of climbing Mt. Everest but if they had only ever rode a helicopter to the top, the knowledge of climbing it would not exist, and surely that is meaningful knowledge given the risks.

So if we rely on AI for proofs I think we lose a sense of what is difficult and why. We lose a sense of what human achievement is. Surely climbing Mt. Everest means more than taking a helicopter up? For students, why bother grinding through all the material of climbing Mt. Everest and then attempting it if the helicopter ride is how things are done now? This would have the affect of destroying knowledge.

(please do not nitpick the analogy because it's the best but perhaps a clumsy way to describe my thoughts)

The current wave of AI slop mathematics might end up driving the next generation of mathematicians away from the subject for the same reason that Mochizuki would have convinced me to quit if his proof had been accepted by the community. Luckily, my professors had the taste to immediately recognize that it was garbage.

This also sounds like a vector for trolling the community with complex putative proofs hiding a known flaw.

I think a better comparison is: mathematics just becomes like mining bitcoins.

I'm not a fan of knocking down things that work, however I also find it hard to be against death of the gatekeeping old guard of any industry.

I think math is just gonna have to suck it up like every other industry now. Math productivity is longer out of reach of the average grad student. Like every other industry they are no longer untouchable and are gonna have to adjust to the new way of things or market forces will do what they always do which is refuse to fund ineffectiveness.

I've had to accept that tech/IT will never be the same. Just how it is. You can thrash against it all you want.

Just compare with the comments on https://news.ycombinator.com/item?id=49639408

Those who think it's me or the machine will fail.

Those who realize how much you can accelerate your research with the help of AI will succeed.

This is a very disrespectful way to make a point about acting with integrity.

You should consider that maybe your views on what makes something ethical or moral are not universal -- and that coming to a discussion with the assumption that your position is the only valid one is not conducive to convincing others who disagree with you.

    > Many graduate students (I know) are having a crisis if any of their research worth it? If AI can (or will) do everything, what's the point of doing experiments and all? This will eventually deter a whole generation of curious minded students from research.

we stumbled into a way of brute forcing intelligence with gradient descent.

You must be new here.

I genuinely enjoy doing this, it's really fun to think critically about what an author wrote or how a particular approach works.

But it does make you wonder, why bother? Probably a frontier model could one shot my algorithm in a day or less. It's incredibly depressing. At least I'm not forced to use it now, but I fear I will have no choice after I join academia or industry in the future.

So I started saying that it only made sense to focus on problems whose solutions would be useful immediately. I even emailed my supervisor about it, arguing that our efforts were “pointless” in the sense that AI-related problems were much more pertinent and had to be prioritised.

My supervisor thought I was bonkers. I still have the email, though. Quoting myself from April 2015:

> By 2030-2040 we will have enough computing power to simulate a human brain neuron by neuron. Once we manage to create a human intelligence we will be one little step away from super intelligence: just set the intelligence to modify itself and see the exponential growth in action. Our human intelligence is bounded by a number of biological factors (e.g. size of a skull) and even the smart human who has ever lived will appear to be a primitive ant to a supper intelligence (machine intelligence will also have perfect motivation). There is plenty of literature on this if you are interested in discussing this further. > > What does it have to do with research in pure maths? I can say that research in pure maths which won't come handy in the next 60 years is just wasted effort. The super intelligence will be able to do maths way better than humans. I believe a lot of current efforts should go into researching of artificial intelligence (or areas to do with AI) instead rather than the pure maths. I want to be proven wrong but most mathematicians I interact with are too narrow-minded to counter me and they just laugh about even contemplating the above. Frankly I am myself so perplexed that I take the above seriously, but I do and it's hurting my motivation.

I'm quite curious what my supervisor thinks of that email now.

And then to get retorts of "it's just just rich techies that want to find 'meaning' in their jobs while millions starve in the third world". Why would we expect the most wealth concentrating technology in history to lead to mass benefits for those in the 3rd world?

It's more about bypassing the culture and processes mathematicians have developed that lead to human understanding, generating new ideas, and bringing up new generations of mathematicians. (See also his article about "non-renewable mining" of good problems.)

Reducing mathematics to "let's just generate results through an isolated and automated system" is a misalignment since it bypasses those processes.

This goes much broader than mathematics or academia. This is the entire basis via which society distributes its wealth: based on a labour market derived valuation of ‘contribution’.

Markets defined entirely by law have distorted our collective understanding of what can actually be built with the knowledge our species has accumulated thus far. How will traditional shields that have protected capital accumulation in tech to survive in a world where governments now realize control of technology is a national priority? Especially as we see its impact on modern warfare, and that such conflict looks like it’s only escalating over time.

Mathematicians appear to me (as an outsider) to exist in a field without such distortions, and I think offer engineers a preview of what’s to come. I certainly have completely ceased sharing original ideas online at this point.

it feels like an unintended consequence of the millennium prize is that people view the [last contributor to the solution] as the only one to make progress on the problem. I've never viewed Poincaré as solved by one person and the objective of the prize was to encourage more people to make attempts and contribute towards progress.

this issue is independent, but in these circumstances perhaps interweaved, with the 'ai is taking over math' concerns

AI companies are investing these resources primarily as a marketing exercise. There is no near term commercial value to a 100 page Lean proof of blow up in an extreme special case of Navier Stokes, besides the bragging rights. As the statement says any commercial value in this stuff comes a very long time later after new insights and techniques have been digested, integrated into the mathematical canon, expressed in ways that don't take a lifetime of study to understand, etc. (things that AI is not yet capable of doing itself). The bragging rights, on the other hand, are massively valuable. There is a mystique to maths that makes "our AI solved a Millenium Prize problem" an irresistable headline for a company like OpenAI.

What the mathematicians are saying is stop pouring resources that most mathematicians can only dream of accessing into projects that are actively damaging to their field. They face a massive challenge of figuring out how maths can evolve in the face of this new technology, and this is not helping.

When there's a discussion about doing something against the damage of the AI industry: "whoopsy, sorry, another cat escape, nothing can be done".

When there's a concrete mention of an actual solution to avoid more cats escaping: "that won't happen, and even if it did, the damage is already done, and in fact it’s not that bad you all just have to go with the future we decided for you."

So the bag is wide open, more cats will escape, and nothing can be done about any of it. not about the ones that got out, and not about the ones still inside. Sounds more like a preemptive excuse for inaction, cosplayed as pragmatism

Photography decimated other forms of visual art, so the concern wasn't wrong. But AI threatens the entirety of human intellectual endeavors. I can make do without oil paintings in my home. I'm not sure I want to live in a future where we make do without brains.

Baudelaire popped up in this article two days ago.

https://www.noemamag.com/a-new-kind-of-creative-poverty/

AI is being treated and pushed as a replacement for every medium.

In a parallel thread omnicognate correctly pointed out that for AI companies it's a direct commercial loss to pour all this money into bruteforcing the solutions to these problems, and that a lot of times the solutions by themselves are not directly commercially valuable. They are doing it for stock price, trying to lure in private capital in preparation for IPOs.

Their models are good, but they are not the moat because Chinese models are good too, so what they are doing, in my opinion, is more harm than good. Mathematics is a science by humans for humans.

There is no doubt that AI has changed the practice of mathematics, just as it has changed the practice of software engineering (and will soon change almost every intellectual job).

Trying to deal with change by saying, "please stop the change" is foolish, IMHO. Mathematicians need to redesign the discipline with AI in mind. But I get that it's easy for me to say that and hard to actually do.

A but like whenever the first sprinter hits a new world record other runners follow along.

Knowing that something is possible tends to strengthen our ability to work with it.

We will potentially see the same with math.

Well yeah... how would there be fewer??

But the point itself is silly. Few people are putting effort into Maths for the fun of it (and of those that are many derive fun from being the only one who can produce a solution). Chess differs in that it never had any point but the game its self.

But there's still play. There's still curiosity. And there's still the drive to understand something for yourself.

But now, LLMs can generate hundreds of books per hour. They make up 80-90% of new arrivals in many nonfiction categories on Amazon. They short-circuit the system, allowing their "authors" to extract money from the system with zero effort by crowding out human work. And it's not even the question of whether these books are good or bad (although overwhelmingly, they're terrible). It's whether it's actually accomplishing anything worthwhile, or just destroying incentives for humans to write or go into any other sort of intellectual work.

In fact, I see many professions push back. Artists, writers, now mathematicians. And I'm amazed that our profession doesn't and that we have so many people who are hooked on vibecoding. I'm still waiting for that 10x payoff. All this velocity and somehow, the landscape of the software I want to use still looks the same as it did in 2021.

Is the fear post-apocalyptic in nature ? We need some human priesthood to carry on tradition ? why ?

Let's assume in the next decade GPT-7 PRO Ultra is cheaply ubiquitous, inspectable, reproducible, transferable, reasonably un-constrained by any institutional interests.

What say the 1% ?

Just a thought.

The hopeful note is that I do think we are entering a golden age for the curious casual/semi-pro mathematician and for niche mathematics areas that won't get the attention of the top labs. Everyone is sprinting to solve the millennium problems, but this is a very exciting time to be in a sub-sub-field where you and 4 others are keeping things alive.

[1] https://substack.com/home/post/p-214740151

Huh. Weird. This hasn't been my take of mathematicians at all. The dozens I know are quite humble and dedicated to math and the beauty one finds in it.

Yes, this has gone so well

Well, one thing is stopping them. There will be no more adoration for their genius.

If you truly do it for understanding and not the attention, carry on. AI should change nothing about your motivations.

Are we really going to take what they say in public seriously?

Ironic or what. Mr Tao may be remembered as Mathematics' Canute.

AI companies are alienating the communities they serve. Instead of a win-win dynamic, they are keen on a win-lose proposition. You dont win trust by one-upping your customer. This is unfortunate and suggests a lack of adults in the room. It also reeks of hubris and is all good when making profits is not a concern. But watch the narrative shift when there is an AI slowdown which is inevitable.

It's not necessarily clear that this statement requires infinity, if you're willing to treat "... =" as a shorthand. You might prefer something like "1 + 1/2 + 1/4 + 1/8 ... -> 2" if it's more clear, where "->" means something like "gets as close as you like without ever getting further away than that", but really the "=" sign is already overloaded in all sorts of subtly different ways anyway, so there's not really any trouble using it here. Almost any rigorous definition you can write down of exactly what that statement means would not rely on the use of infinity.

A better example would be a limit that equals sqrt(2) which finitists would probably say cannot represent a real object because it is only defined as the end of an infinite process.

Would you argue it didnt benefit humanity, because taxi/bus drivers are nolonger required

Why? If someone makes an innovation that undercuts the underpinnings of some existing institution, why are they are responsible for cleaning up its failure?

Everything is sensationalized, every super niche happenstance is sold as earth-shattering drama, the outrage arms race is so tiresome.

"Everybody's enraged, why don't you like this unethical thing being done to you by a company?"

What's going on here?

That's like saying that programming is about producing valid programs in various programming languages.

But then who decides why a statement is mor important than another? In the eyes of a formal systems all statements are born equal.