I know why the mathematicians weep (derekthompson.org)

I feel like we should be doing everything we can to not revert back to that way of life.

Industrial and technological progress has birthed better conditions overall, but at a cost of locally worse ones. Mine workers that got displaced didn’t end up being machine operators - most of them landed on benefits or sporadic gig work. Only the next generation was able to be properly trained and educated to be able to benefit from the technological advancement en masse. And we might see that happening again.

I don't buy this argument, much like I don't buy the similar "they said the same when calculators were introduced".

It negates that some changes are qualitatively and scale-wise different. Also, by gradually chipping away at things that humans do, you (we) keep shrinking and shrinking until there's nothing left, every step of the way some people claiming "well, the telegraph operators said the same". Keep in mind that, while it's hyperbole, some AI entrepreneurs are actually saying stuff like "maybe jobs will become a hobby". Unrealistic for now, maybe, but if they do become a hobby, what leverage will we have at all?

In other words, it's true the past is worth keeping in mind, but not to the exclusion of the possibility that this time it's different. Or that the trends of the past actually point to harm in the near future.

When you talk with machines, you "outsource" that central human society-building element.

But machines never socialize back in any meaningful way. You gradually lose yourself.

I do too, but it means I have more time in my life to spend with people.

This is a general issue that is being forced on us by AI companies; AI is not a human being, it is a tool that we can use to do things. We don't have them as coworkers or subordinates, they're objects specialized in language processing. Thinking otherwise leads people to all kinds of existential and psychological issues that in the end make no sense.

This is why I see myself leaving dev within a year. Remote working is enough of a embuggererance for my mental health and I've be unhappy in work much of the time since 2020 largely due to that, AI will finish me off.

Career suicide and going of to wait tables, or be a hospital porter, or anything else with people even if I'm going to hate those people, with the massive pay cut that entails, will be better than actual suicide after a couple more years trying to stick at it.

There came a point a few months ago where I realised that I had to either accept becoming obscelete or change my while outlook on using AI and trying to catch up. I didn't choose the AI option.

I'll keep that in mind. If I'm suffering more than you are, the solution is to bring you down to my level.

> Not for those without status (which is most people on this planet)

Look at how coal miners react to the idea of getting rid of coal as an energy source; whether or not you think they have prestige or status, there's certainly an amount of dread there that I'd expect to somewhat resemble the type we're talking about here.

This is what you don't get. _I_ already know it's not worth it. I've always known. It's everyone else! It's a wife who cares whether or not her husband has a good career. It's the police who care if you look like you're poor. It's the way every single person in society obsessively cares about the status of others.

What in the world makes you think I've had to build up status for my own sake? It's absolutely baffling that anyone could be confused about this.

Superintelligence might even help.

So how do you plan to maintain food, clothing and shelter when you have no leverage?

For the low entry fee of ~$200 a month you too can escape the permanent underclass - which is more than some countries average citizen makes in an entire year.

Even if we vastly increase productivity beyond our current wildest dreams, the production system takes time to adjust to changes in demand, creating continuous points of scarcity.

But Georgist economics are way ahead of how to deal with this. Tax the ownership/control of things that can not be increased (e.g. land).

What we have is a DoS in research verification.

Both depend on Logic, which is where "deontic scorekeeping" comes in. You can only establish something (score) is true and believable by 'keeping score' and having some level of understanding about the language or math you are reading.

The article is mostly referring to the connection that comes from shared understanding between humans, particularly mathematicians, as they build new shared understanding, and realize there are limits to what they can know and 'leave behind' logic for others to pick up and carry on.

That connection has been severed, or strained a bit, but I'm sure new ones will be created. It just might be of the same flavor that mathematicians prefer, therefore they weep.

And if I understand "deontic scorekeeping" correctly, these forums are a better representation of the concept. You will lend some amount of belief to what is said/read based on what you know.

What an LLM can't do (at least with the current architecture) is sustain a commitment that it can be held accountable for later. It doesn't have durable "perspective". It can't update its priors. Maybe a new architecture will come along that gives LLMs memory plasticity, at which point they will shift into being individuated quasi-persons, but that's not what we've got now.

I've written some about this here, if you're interested: https://elliotmilco.substack.com/p/deontic-scorekeeping-comm...

I suppose we would see some interaction with AI in that sense, but it'll take place over the course of learn cycles and when AI starts ingesting its own ideas it typically goes wrong somewhere down the line.

Likewise OpenAI doesn't have "skin" of any shape or value in e.g. the mathematics community or even in mathematics research per se. In my cynical view, they are exploiting the exposure of mathematical research to automation in an effort to create a case study of their product obsoleting a highly visible and well-leveraged and relatively small profession. Regardless, I doubt even they would say they have durable skin in mathematics in any sense, certainly not in either of the senses I mentioned above. But their output inhabits the same space as mathematicians, who have to contend with the consequences

And potentially steals [1].

[1] https://en.wikipedia.org/wiki/Navier%E2%80%93Stokes_priority...

And yeah, consciousness based arguments are bunk IMO. It's clear that LLMs are in a basic sense "aware", and even to some extent "sapient" (in Brandom's usage), i.e. they can justify claims and object to things at least locally. I follow Rorty on the question of consciousness, which is to say that consciousness has no deeper meaning in practice beyond "are you awake", and in philosophical discussion it's largely a proxy for "are you one of us" in some amorphous sense.

[1] https://sites.pitt.edu/~rbrandom/Courses/2022%20Phil%20of%20...

Edit: One of my ways of benchmarking new AI models is actually to type a version of this prompt in and see how sophisticated the response is: "Talk to me about the concept of deontic scorekeeping in the context of llms". You'll get good mileage out of that.

I have to wonder if this is the equivalent of teleporting back in time and finding Marco Polo or any of those early explorers who traveled for the love of the game (and fame and money), and opening up Google Maps on your phone and showing them the zoomed out world. And showing them any island or village they could name.

Computer can iterate without frustration on a completely random garbage. Once in a while it stumbles upon a gem. Even if it's one in 10000, that's still hundreds of good ideas in a day (provided enough GPUs).

I don't think this matters a bit. We individually can't demonstrate "consciousness" in other humans, it's not even a word with a definition. What we can do is miss the feeling of discovering something on one's own rather than having it explained to you over and over again until you understand it.

I mean, dead mathematicians are definitely not "conscious," but their books keep you from discovering things that you may have discovered yourself if left to your own devices.

The joy that was mentioned is the joy of knowing a bit of gossip that no one else knows, of being the first explainer. Interpretation is just being able to pass gossip on; a decidedly lesser joy. And there's no reason that AI shouldn't be able to explain findings in an intuitive way - LLMs at least purely operate based on human intuition as written in human communications and then captured in matrices.

If these proofs are good (doubtful, imo) and can be reduced (intuitively distilled, sorry), there's no reason that an LLM shouldn't be able to do that.

In many ways, the teleporting machine that teleports to the solution is against the point of the questions. No one wants the solution to be "Run $10,000,000 worth of tokens to cover all possible Lean formulations of the problem".

I think for Mathematicians, I'd say that the human level formulation is still possible and with a new perspective. We as a society (and outside of Mathematics) need to also understand that this AI teleport to the answer thing has less value than we all expected. The value is in the journey.

The open question is if people will still want to take the journey if the teleport machine exists. Or if the existence of the teleport machine halts exploration.

Saying we understand training and deploying machine translation systems that have strong predictive (rather than descriptive) power, apparently has very little overlap with defining a formal, analyzable, mathematical model of language. Mathematician who refer to "understanding" generally mean the second thing, but the first thing is also a kind of understanding. Just messier.

Why?

Or maybe it can; the early image generators famously did output horrific images sometimes because of their limitations, but these outputs qualify nonetheless creations.

Now the interesting thing is that it is humans who evaluate them as horrific, worthless or interesting.

Certainly AIs could pretend they have that sort of opinions and judgement, but hopefully humans will remain more interested in the opinions of other humans.

At the same time, I think it's pretty amazing how this is the fundamental question of AI's trajectory, yet people on both sides seem so sure of their viewpoint with little actual evidence or data to back it up.

That is, it's possible that LLM-based approaches will be forever limited to this "convex hull" of current training data, and thus humans will always have a role to play (and will retain primacy when it comes to overall intelligence), because it appears that humans are not limited to this convex hull based on previous discoveries (e.g. I think basically every scientist would argue Einstein's general relativity was outside the convex hull that existed at that time).

Yet, if LLM-based approaches are eventually able to surpass humans on this "novel discovery" axis, then we as a species are basically done for when it comes to intellectual primacy - and then it becomes a very strange and scary world. And many AI researchers at the top companies clearly believe current approaches will scale and eventually (pretty soon I'll add) lead us to RSI.

I have no idea which side is right, and I'll admit I really hope LLMs eventually hit a wall. But again, I just am surprised that many seem to be putting forth their beliefs, but I would love to know if there is any good evidence one way or the other (or better yet, a more concrete and verifiable definition of the "convex hull" so that researchers could provide some falsifiable examples).

In my undergrad math department, only 2-3 faculty members out of a dozen bothered with grants.

AI is going to suck hard for lots of professions, software engineers and mathematicians included. I think it's much more productive for folks in these professions to show solidarity with each other given the way AI will disrupt their livelihoods, rather than showing disdain based on who you think may have it slightly better.

I know this intimately because I identify, roughly, as a programmer first and security researcher second. Guess which two fields ai companies decided to specialize in first? Yes, the FML ones.

The intellectually rewarding part of my job/skills - the only reason I enjoyed them - are, not completely gone yet, but threatened. And shrinking. This is a very scary observation.

I know from ‘isomorphic situations’ lessons in history that fighting the tide is pointless. I have to decide intelligently how to respond. Easier said than done when your livelihood and identity are at stake.

For now I still have a job but I and everyone around me is worried. In a year, At best our jobs will be very different.

So yes I empathize with these mathematicians enormously.

It's not fun to think of a future where knowledge workers become irrelevant in society and we're back to just physical work and managerial elites.

Just like in past history.

Yes, it sucks, for software engineers, those in the past, and those in the future. As long as you and others fall for the straw man perhaps you and others will get more and more tricked and hurt, misplacing your aggression and hatred on each other rather than those receiving most of the extracted value.

The economist John Keynes predicted almost 100 years ago that technological advances would lead to a 15 hour workweek. If AI now makes you 5x as productive, then why aren't you working 1 day a week for the same pay? Why didn't Keynes' vision come true.

Are phones an "automation of existing jobs"? Maybe couriers (??), but most of what people use them for goes far beyond that.

What about the software used to drive trains or autopilot planes? We still have conductors and pilots, but now the outcomes are safer on average and more efficient.

Or something like Google Maps -- all that replaced was the paper booklet people used to keep in their cars, it's not like we used to employ 'navigators' to go out on a road trip.

Or something like Intuit QuickBooks: there's still an accountant in the loop, and if anything more accountants today than there were a few decades ago.

Certainly some software is essentially automatory in nature. But to characterize 'nearly all' as being such just betrays a lack of awareness for the rest of the field.

> That one is somehow deserving of sympathy and that of mathematicians is not

He didn't say that mathematicians aren't deserving, he was griping that people said this about SWEs. Do you think mathematicians are deserving but SWEs aren't?

"Ants and bees can also work together in huge numbers, but they do so in a very rigid manner and only with close relatives. Wolves and chimpanzees cooperate far more flexibly than ants, but they can do so only with small numbers of other individuals that they know intimately. Sapiens can cooperate in extremely flexible ways with countless numbers of strangers. That’s why Sapiens rule the world, whereas ants eat our leftovers and chimps are locked up in zoos and research laboratories."

(...)

"You could never pack a thousand chimpanzees into Times Square, the United Nations, the Vatican or the New York Stock Exchange. You would get pandemonium. By contrast, thousands of Sapiens regularly congregate in such places, and they usually produce orderly networks and trade rather than violence and chaos."

---

And the passage I recalled from memory is actually from a TED Talk ("Why humans run the world", 2015 - https://www.youtube.com/watch?v=nzj7Wg4DAbs&t=231s)

"If you try to crowd 100,000 chimpanzees into Wembley Stadium, or into Oxford Street, or into a church, or into a school, you will get complete chaos. But you can put 100,000 humans together in Wembley Stadium, and they will usually behave reasonably well, watch the match together and sing along."

---

But I’m not giving up on your celebration of our curiosity.

Speaking of curiosity, I’d bring Peter Gray and his "Free to Learn" book into the conversation.

Take a look at the second of the five characteristics of play that Peter Gray proposes:

"Play is activity that, from the conscious perspective of the player, is done for its own sake more than for some reward outside of the activity itself. In other words, it is behavior in which means are more valued than ends.

When people are not playing, what they value most are the results of their actions. [...] When people are not playing, they typically opt for the least effortful way of achieving their goal.

In play, however, all this is reversed. In play, attention is focused on the means more than the ends, and players do not necessarily look for the easiest routes to achieving the ends. Play often has goals, but the goals are experienced as part and parcel of the activity, not as the primary reason for the activity. Goals in play are subordinate to the means for achieving them."

Using Gray’s definition, mathematicians might actually be mourning the loss of play — the process itself, the thinking, the exploring — rather than "just" reaching the final result (via ChatGPT).

Is this a valid way of looking at it through Peter Gray's lens?

There were mistakes that could easily be seen with a diagram rendering three big results wrong, exposition is terrible, no coherent underlying theory generated from solving so many problems, as Thomas Bloom pointed out even duplicate proofs of the same statement in a single paper.

It clicked for me, "this is slop but slop can be superhuman and useful!". Similar to programming where it can generate as many criticisms of your codebase but you have to weed them out.

LLMs haven’t done much in the way of informing me or others I know regarding “who and how many”.

The recent barf-out done by LLM companies is useful if what are you are interested in is an unreadable mess and someone giving you the final answer. You might be able to extract some new "aha!" moments out of it, but we will have to wait and see what happens once people go through it. I would hypothesize the LLM output will be good at hinting at some non-obvious relationships and patterns, but I doubt it will yield anything of substance in its own right.

Historically, proving difficult theorems required mathematicians to sometimes invent new fields, new ways of looking at problems, the discovery of which itself yielded incalculable contributions to both mathematics and other fields. It is this that is the greater contribution of and to mathematics.

What is true, is that chat models (and by extension agents) have a very characteristic style after their LLM went through post-training. But the pre-trained language model doesn't have that style.

So if you want to write well with an LLM, you could and it is a well understood problem. It is more a matter of that not being of much interest, as it competes with having the model be good for chat interfaces.

How to have a well-aligned, RL'ed, safe or thinking model, which also writes well in long-form, is a bit an open problem. Though presumably agents could do it when they have access to a pretrained model.

I read this passage and thought, "has Derek even talked to software developers"?

That feels to me like the first discipline to go through this, where their daily activities are massively changed by GenAI.

That is why it's so disturbing. After several paragraphs, you realize there's nothing human there even when the actual information is present. People that do create AI piece are OK with it because they already have the context, but for the reader, it's always the uncanny valley.

Their math is as just as good as their rambling writing and their Ghiblified poster art. Most people don't seem to realize that.

It's not at all extraordinary if you understand how they're trained. For math and programming it's possible to generate an infinite amount of training data with correct answers, as compilers and theorem provers can guarantee correctness. For writing, the quality is not objective, it's determined by human preference, so you need humans in the loop, which slows things down massively and limits the amount of data that can be generated.

https://en.wikipedia.org/wiki/Life,_Liberty_and_the_pursuit_...

To me it begs the question: in a theoretical world where we don't have human mathematicians anymore, what is the point of these proofs if there's no human who actually understands it?

If the point of a lot of this math is only to expand the boundaries of human knowledge, it all seems kind of pointless.

For some reason I'm stuck in a real depressing, doomer algorithm / homepage on Twitter. But everyone on my homepage seems to have a take on the HNer comments along the lines of "see, this is why you don't make your job into your identity".

But that's such an antihuman response. The only reason you can put decades into a field to grind and build expertise is because you have an identity-level interest in a subject.

It's also not a good thing if the optimal way to live going forward is to have no passion for anything lest it changes so that we won't feel bad. I don't think that's possible unless you remove the human part of our brains or make life so horrible that we're happy just to survive another day.

And I can’t imagine feeling the same about any group of people. It’s been odd, to me. I’m just trying to support myself and the people in my life, and now there’s a looming fear that I can’t do that anymore… business owners are drooling over that, and other sectors are laughing.

Hard times, I guess.

If AI got significantly better at performing surgery overnight and doctors started crying about how they were going to be replaced, what do you think the public response would be? Non-doctors want the best possible care; they aren’t interested in protecting an industry. The same will be said for pilots and lawyers, too.

People dislike self-serving whingeing, end of story, but that’s doubly true when it comes out of a profession (theoretical mathematics) that is mostly practically useless.

when mathematicians complaining about losing the part of the job that they love, people think "if you love doing it, why is anybody paying you? you're describing a hobby."

They are showing as much care for mathematicians as they received themselves, which is not much.

There is little or no time, or incentives for care as described by the blog.