Wow, this is more existentially depressing and terrifying than any x-risk doomer prediction of grey goo and engineered viruses. You got a knack for that, Trace. Don't let it be chabuduo'd away.
I predict that in a generation or two, the few experts standing will be the people absolutely obsessed with the process/love of the game. You know, the weirdoes who put in the thousands of hours of reps because they enjoy it (while most people consider it a slog).
I learned how to write as a kid because I liked writing. I couldn't *not* write. I'm sure lots of artists, as kids, couldn't *not* draw all the time. They'll be the last holdouts of expertise.
That's a good point. There are plenty of artists out there doing it for love of the game rather than making money. That said, it's one thing to know there are some people who are better than you, will it change anything to know that AI is too (and therefore anyone, not just the best can do better than you with just a few seconds effort)
Hmm, maybe I should use an AI to write my comment here, it might be the proper response.
"ChatGPT, please write a comment of around 500 words about the idea that technological changes always have people bemoaning the supposed damage it will do to the human intellect. Reference that Socrates complained about the mental degeneration which would be brought about by the peril of the newfangled "writing". Include also the moral panic of the dangers of "calculators" and similar."
I'm not sure if that would be proving, or disproving, the point.
Every time people talk about Socrates and writing, I think about how every few years people rediscover the lost Greek art of "mnemonics" and marvel at all the clever memory techniques the ancients knew. Every new wave of technology brings real benefits, but doomsayers are very often correct on their narrow concerns.
Ah, but there's a big difference between people aren't going to be memorizing lengthy epic poems anymore, and society will mentally degenerate. I mean, if you go around today saying students have gotten so much dumber because they can't recite the epics from memory anymore like the Good Old Days, having been crippled by the stunting use of "writing", you will not be acclaimed as having produced much social insight. This holds even though it's in fact true that essentially people can't recite the epics from memory anymore. The key part is that this is not considered an indicator of how bad the world has become since the speaker was a child.
Very very few people nowadays have ever hunted and killed their own meat, which was a key human skill for a long time (far more important to humanity overall than epic poems). And they would be totally lost if they suddenly needed to do so due to civilizational collapse (n.b. guns and ammo become scarce then too - you need to make a bow and arrows from wood). While there's a bit of survivalist-style worry about that, the loss of skills of hunting and trapping are generally not considered a tragedy - even though it's a major upheaval.
The fragility and inadequacy of my knowledge to sustain my life (especially at current comfort levels) always hits me. The British guy who tried to make a toaster drove it home.
A tangent off that point is in most of these domains, I’m not substituting being an expert for not being an expert. It’s hiring an expert vs using an LLM. I was never going to be expert graphic designer, front end developer, or drywall hanger.
I’m not convinced we’ll lose our desire to be good at stuff. We already weren’t good at most stuff anyway. We may be more shaped by what the learning and expertise will make us than what we have to do to survive or live comfortably.
My impression is that often contemporary predictions about the downsides of new technology were completely accurate. In many cases we have lost even the memory of what we have lost. When recorded music was introduced, people accurately predicted that it would largely be the end of amateur music-making, which had up to that point been a quotidien family activity.
The key part is "supposed damage it will do to the human intellect". See my later comment about how almost everyone nowadays, with just a few exceptions, cannot hunt to save their life. And making a fire is a rare art. This is not a good argument against technology.
"The process of skill development is laborious, frustrating, and often humiliating, because at the beginning, learners are far from “chabuduo.” They are simply 差 (chā): falling very short indeed."
This is all painfully true. I work in legal marketing, particularly in blog writing, and so many of our recent hires have let AI take over their writing AND their thinking. We’ve lost formerly decent employees whose brains turned into AI oatmeal. The unfortunate reality is that, in our profession at least, AI can make so-so writers good enough that they can get the job done, and most people can’t tell the difference.
Did the “cha” overlap between “ChatGPT” and “chabuduo” help spur this article or is that just a cherry on top? Obviously I know the Chinese pronunciation is something way different, but to a Chinese-nonknower like myself, I even saw the “duo” as adding to the wordplay.
re. "most people may never develop most intellectual skills past a rudimentary level, relying instead on good enough tools to create things skilled professionals might cringe at but that are far outside the capacity of novices to produce in any given domain.":
My experience is that chatbots now discourse with knowledge, reason, and open mindedness beyond that of all but a small fraction of people. most people have never been exposed to fair argumentation on that level before. chatbots especially outclass people in that they have no sacred beliefs; they can be swayed if your evidence and reasoning is solid. they will admit it immediately if you point out factual or logical errors in their argument. So I think they will improve the reasoning of people who use them, just as a good teacher, who marshals evidence and admits errors, does so even while lecturing.
Lovely article, Jack. But for fun and between friends, I would happily take the contra position.
Contra to the claims of the piece, I expect chess experts to get much better at chess. I expect artists to get much better at art. In general, I expect human skill to flourish. After all, there is no obvious reason why it wouldn't.
Furthermore, the idea of AI-prompting being distinct from human skill will rapidly become anachronistic as all humans on Earth adjust quickly to the style and output of work which is easily created using popular models.
It will be increasingly easy to judge the skill of an expert AI artist as compared to an amateur AI artist, the same way it is easy to judge the skill of an expert watercolor painting from an amateur one.
AI will be deemed a valid and reasonable tool for creation the same way computing is now. Many competitions will remain which deem it invalid, the same way we have an Olympics where humans race in an era where we have engines that far outpace every aspect of our strength.
For the claims of the article to be prophetic, humans would not only need to perform worse than AI on all trainable tasks (which we will), but also stop innovating (which we will not) AND stop caring about the creation of worthy competitions of human skill (which we certainly will not).
Will humans give up? Will we stop being determined to climb the tallest mountains or sing the most beautiful songs? Will human children lose motivation for self-actualization?
I think not. The human spirit is not so easily crushed, and even if it were, such depressed cultures will not be competitive. Stronger cultures with Asabiyyah will succeed us, and they will be fine. My child will be fine, for instance. Since that's the case, I'm sure she'll have peers, and cultural knowledge now spreads quickly enough with the internet to create new competitive cultures within a single generation.
This is all presuming we navigate technological risks appropriately, of course, but that's out of scope for the conversation about how AI might cause a cultural depression.
> In the past, demanding skills like navigating by the stars or typesetting by hand withered and died as technology provided good enough alternatives. Today, we face the prospect of some form of that happening to every intellectual skill at once.
Worse: in the past those skills whithered because technology provided better than them alternatives (e.g. compass or GPS vs navigating by the stars), even when considering the best practicioners of those skills.
AI, todays, provides merely "good enough", but not as good as the best practiocioners, and often not as good as average practiocioners either.
I don't agree about the AI-header pictures. First, it's plagiarism. Yes, the same is true of coding, as you point out, but plagiarizing the work of artists, who as a class don't have a reputation for living or ever having lived in any kind of material comfort, seems cruel.
Secondly, AI for code seems more like replacing a tool with a better tool, not withstanding the few software engineers who do find code to meet some standard for aesthetic beauty. Personally, as someone who has done a decent amount of scripting in MATLAB, I'm happy to totally forget that skill, because I'm still working 40 hours a week in a job that requires critical thinking. My skills in MATLAB have been devalued, but I don't think anything of aesthetic value has been really lost. AI for visual art seems like... I don't know, words fail me, but just somehow really wrong. In the small amount of blogging that I have done, I use Creative Commons Wikimedia pictures for my headers, and I always credit the photographer or artist.
This was a brilliant article! And in some way, it had to be lol. I'm curious about something completely unrelated, if you don't mind. Is the name 'tracing woodgrains' a reference to Orson Scott Cards Ender series?
This is a good insight and more or less correct. I am less sure how to evaluate it. Maybe the optimist case that we'll just all have to learn the next level of skills and that's a good thing is true.
When I was younger I learned some things and achieved mastery. Now that I am in the world of work, I am expected to work on complex, ever changing systems that are not worth getting mastery of. Some, but not all, of the nuts-and-bolts of what I do depends on the old masteries (unix command etc).
My real job, even before AI, was at that higher level. But I was slowed down by having to deal with subsystems that I only half understood. That makes the AI really valuable, because it can go and spend twenty minutes doing that grunt work, or even if it spends a day, I don't have to waste my time.
So in this context, the evolution you point to is actually a positive. But I doubt that's the whole story. I don't know how the whole thing will shake out.
I think this accepts the longevity of AI, which is doubtful. The amount of money to run these things, the resources they consume, the HATE they breed, are so intense that I seriously doubt it will continue to exist in the long term.
Sorry, I'm late. Meant to be done with refactoring all of math Notation and be working on the mentat-project already. Looks like I'll have to be doing both in parralel, because the mentat-stuff is becoming rapidly more important. kuai dian, kuai dian :(
Wow, this is more existentially depressing and terrifying than any x-risk doomer prediction of grey goo and engineered viruses. You got a knack for that, Trace. Don't let it be chabuduo'd away.
The grey goo is still there. It's just that, in this scenario, it's the final state of the contents of our skulls.
I predict that in a generation or two, the few experts standing will be the people absolutely obsessed with the process/love of the game. You know, the weirdoes who put in the thousands of hours of reps because they enjoy it (while most people consider it a slog).
I learned how to write as a kid because I liked writing. I couldn't *not* write. I'm sure lots of artists, as kids, couldn't *not* draw all the time. They'll be the last holdouts of expertise.
That's a good point. There are plenty of artists out there doing it for love of the game rather than making money. That said, it's one thing to know there are some people who are better than you, will it change anything to know that AI is too (and therefore anyone, not just the best can do better than you with just a few seconds effort)
Hmm, maybe I should use an AI to write my comment here, it might be the proper response.
"ChatGPT, please write a comment of around 500 words about the idea that technological changes always have people bemoaning the supposed damage it will do to the human intellect. Reference that Socrates complained about the mental degeneration which would be brought about by the peril of the newfangled "writing". Include also the moral panic of the dangers of "calculators" and similar."
I'm not sure if that would be proving, or disproving, the point.
Every time people talk about Socrates and writing, I think about how every few years people rediscover the lost Greek art of "mnemonics" and marvel at all the clever memory techniques the ancients knew. Every new wave of technology brings real benefits, but doomsayers are very often correct on their narrow concerns.
I write more about this in my review of Amusing Ourselves to Death: https://www.tracingwoodgrains.com/p/book-review-amusing-ourselves-to
Ah, but there's a big difference between people aren't going to be memorizing lengthy epic poems anymore, and society will mentally degenerate. I mean, if you go around today saying students have gotten so much dumber because they can't recite the epics from memory anymore like the Good Old Days, having been crippled by the stunting use of "writing", you will not be acclaimed as having produced much social insight. This holds even though it's in fact true that essentially people can't recite the epics from memory anymore. The key part is that this is not considered an indicator of how bad the world has become since the speaker was a child.
Very very few people nowadays have ever hunted and killed their own meat, which was a key human skill for a long time (far more important to humanity overall than epic poems). And they would be totally lost if they suddenly needed to do so due to civilizational collapse (n.b. guns and ammo become scarce then too - you need to make a bow and arrows from wood). While there's a bit of survivalist-style worry about that, the loss of skills of hunting and trapping are generally not considered a tragedy - even though it's a major upheaval.
The fragility and inadequacy of my knowledge to sustain my life (especially at current comfort levels) always hits me. The British guy who tried to make a toaster drove it home.
A tangent off that point is in most of these domains, I’m not substituting being an expert for not being an expert. It’s hiring an expert vs using an LLM. I was never going to be expert graphic designer, front end developer, or drywall hanger.
I’m not convinced we’ll lose our desire to be good at stuff. We already weren’t good at most stuff anyway. We may be more shaped by what the learning and expertise will make us than what we have to do to survive or live comfortably.
My impression is that often contemporary predictions about the downsides of new technology were completely accurate. In many cases we have lost even the memory of what we have lost. When recorded music was introduced, people accurately predicted that it would largely be the end of amateur music-making, which had up to that point been a quotidien family activity.
The key part is "supposed damage it will do to the human intellect". See my later comment about how almost everyone nowadays, with just a few exceptions, cannot hunt to save their life. And making a fire is a rare art. This is not a good argument against technology.
Wonderful article!
"The process of skill development is laborious, frustrating, and often humiliating, because at the beginning, learners are far from “chabuduo.” They are simply 差 (chā): falling very short indeed."
In fact they might even by "cha hen duo" 差很多!
This is all painfully true. I work in legal marketing, particularly in blog writing, and so many of our recent hires have let AI take over their writing AND their thinking. We’ve lost formerly decent employees whose brains turned into AI oatmeal. The unfortunate reality is that, in our profession at least, AI can make so-so writers good enough that they can get the job done, and most people can’t tell the difference.
Great article! Really crisp framing of a worrisome possible future.
I believe that humans will choose to hold on to our agency though, and wrote up an essay arguing so: https://www.noidlesitting.com/p/we-need-not-sleepwalk-into-twilight
Did the “cha” overlap between “ChatGPT” and “chabuduo” help spur this article or is that just a cherry on top? Obviously I know the Chinese pronunciation is something way different, but to a Chinese-nonknower like myself, I even saw the “duo” as adding to the wordplay.
Pure coincidence; no wordplay. I’ve had the reference in mind for a while.
re. "most people may never develop most intellectual skills past a rudimentary level, relying instead on good enough tools to create things skilled professionals might cringe at but that are far outside the capacity of novices to produce in any given domain.":
My experience is that chatbots now discourse with knowledge, reason, and open mindedness beyond that of all but a small fraction of people. most people have never been exposed to fair argumentation on that level before. chatbots especially outclass people in that they have no sacred beliefs; they can be swayed if your evidence and reasoning is solid. they will admit it immediately if you point out factual or logical errors in their argument. So I think they will improve the reasoning of people who use them, just as a good teacher, who marshals evidence and admits errors, does so even while lecturing.
Lovely article, Jack. But for fun and between friends, I would happily take the contra position.
Contra to the claims of the piece, I expect chess experts to get much better at chess. I expect artists to get much better at art. In general, I expect human skill to flourish. After all, there is no obvious reason why it wouldn't.
Furthermore, the idea of AI-prompting being distinct from human skill will rapidly become anachronistic as all humans on Earth adjust quickly to the style and output of work which is easily created using popular models.
It will be increasingly easy to judge the skill of an expert AI artist as compared to an amateur AI artist, the same way it is easy to judge the skill of an expert watercolor painting from an amateur one.
AI will be deemed a valid and reasonable tool for creation the same way computing is now. Many competitions will remain which deem it invalid, the same way we have an Olympics where humans race in an era where we have engines that far outpace every aspect of our strength.
For the claims of the article to be prophetic, humans would not only need to perform worse than AI on all trainable tasks (which we will), but also stop innovating (which we will not) AND stop caring about the creation of worthy competitions of human skill (which we certainly will not).
One way to underscore my general stance is this:
Will humans give up? Will we stop being determined to climb the tallest mountains or sing the most beautiful songs? Will human children lose motivation for self-actualization?
I think not. The human spirit is not so easily crushed, and even if it were, such depressed cultures will not be competitive. Stronger cultures with Asabiyyah will succeed us, and they will be fine. My child will be fine, for instance. Since that's the case, I'm sure she'll have peers, and cultural knowledge now spreads quickly enough with the internet to create new competitive cultures within a single generation.
This is all presuming we navigate technological risks appropriately, of course, but that's out of scope for the conversation about how AI might cause a cultural depression.
We'll develop new norms. Probably platforms will auto-flag AI text.
Or maybe nobody will care, after enough time.
> In the past, demanding skills like navigating by the stars or typesetting by hand withered and died as technology provided good enough alternatives. Today, we face the prospect of some form of that happening to every intellectual skill at once.
Worse: in the past those skills whithered because technology provided better than them alternatives (e.g. compass or GPS vs navigating by the stars), even when considering the best practicioners of those skills.
AI, todays, provides merely "good enough", but not as good as the best practiocioners, and often not as good as average practiocioners either.
I don't agree about the AI-header pictures. First, it's plagiarism. Yes, the same is true of coding, as you point out, but plagiarizing the work of artists, who as a class don't have a reputation for living or ever having lived in any kind of material comfort, seems cruel.
Secondly, AI for code seems more like replacing a tool with a better tool, not withstanding the few software engineers who do find code to meet some standard for aesthetic beauty. Personally, as someone who has done a decent amount of scripting in MATLAB, I'm happy to totally forget that skill, because I'm still working 40 hours a week in a job that requires critical thinking. My skills in MATLAB have been devalued, but I don't think anything of aesthetic value has been really lost. AI for visual art seems like... I don't know, words fail me, but just somehow really wrong. In the small amount of blogging that I have done, I use Creative Commons Wikimedia pictures for my headers, and I always credit the photographer or artist.
This was a brilliant article! And in some way, it had to be lol. I'm curious about something completely unrelated, if you don't mind. Is the name 'tracing woodgrains' a reference to Orson Scott Cards Ender series?
It is. I believe one of his first articles on substack explained it. Hell of a series, i still return to it every couple of years.
This is a good insight and more or less correct. I am less sure how to evaluate it. Maybe the optimist case that we'll just all have to learn the next level of skills and that's a good thing is true.
When I was younger I learned some things and achieved mastery. Now that I am in the world of work, I am expected to work on complex, ever changing systems that are not worth getting mastery of. Some, but not all, of the nuts-and-bolts of what I do depends on the old masteries (unix command etc).
My real job, even before AI, was at that higher level. But I was slowed down by having to deal with subsystems that I only half understood. That makes the AI really valuable, because it can go and spend twenty minutes doing that grunt work, or even if it spends a day, I don't have to waste my time.
So in this context, the evolution you point to is actually a positive. But I doubt that's the whole story. I don't know how the whole thing will shake out.
I think this accepts the longevity of AI, which is doubtful. The amount of money to run these things, the resources they consume, the HATE they breed, are so intense that I seriously doubt it will continue to exist in the long term.
Sorry, I'm late. Meant to be done with refactoring all of math Notation and be working on the mentat-project already. Looks like I'll have to be doing both in parralel, because the mentat-stuff is becoming rapidly more important. kuai dian, kuai dian :(