Yes, I agree with most of this - AND IT'S GOOD! (mostly). What you didn't mention in this post is the Industrial Revolution. When humans learned how to harness mechanical power on a large scale, it changed the entire face of civilization. The shift from agrarian to manufacturing economy was massive, with extensive population displacement from rural areas to cities. I think that's the way to look at things. Not Doomers or Singularity hypesters. I'm firmly in the camp of "AI Is Normal Technology" - but normal technology can have huge, absolutely enormous, effects on society.
You might want to take a look sometime into what the real, literal, Luddites were talking about (if you don't know already).
To be tedious, note we've been in the future for a while now with a worldwide interconnected network - something that was Science Fiction for a long time.
What should we do? I don't know. I personally regret missing the opportunity to get rich fast in bubble-money. There's always such an opportunity at the start of a technology change, but I think it's past that point now. Learning the details of AI is probably a good bet if you're young programmer type. It's going to be a crowded field, but also I think an expanding one for a while.
In general, I'd advocate that society make sure the social change problems are moderated by sharing the created wealth. But I know how purely aspiration that idea is.
Can you expand on the reasoning that leads you to conclude that "AI is normal technology"?
It seems to me that it is anything but. It is not only the first technology that promises to exceed human capabilities in the one area that humans have always surpassed every other life form on this planet, the one that made us capable of dominating all others from the largest to the smallest, but the first technology that is almost certainly going to be capable of iteratively improving itself, without any human involvement required.
Certainly opposing automation to preserve jobs or the social status quo is unethical- the very real human cost of job displacement is properly solved with a generous social safety net, not by robbing future generations of economic progress.
However, AI isn't just automation. These are minds; not yet as general as human minds, but if current benchmark trends continue for another two or three years, they will be. Introducing an entirely new kind of intelligent labor into civilization is something deeply unprecedented. Our familiar heuristics about the benefits of industrial automation and social progress aren't going to be reliable indicators of what to expect- to predict accurately, we have to look closely at the details.
First, let's assume away ASI- assume that AI progress plateaus around the level of human experts and goes no further (aside from a few narrow fields like coding and math). In this near future, AI is able to pilot cheap mass-manufactured robots and drones as easily as we move our bodies, and will be able to do effectively everything a human worker can, only far cheaper. If this sounds far-fetched, I encourage you to look up the benchmark trends in robotics, long-horizon task completion, sample efficiency and general reasoning- straightforward extrapolation has AI passing the human level at all of these in '28 or '29.
When historic automation replaced a job, the overall demand for human labor didn't decrease- the surplus from automation allows for new kinds of more valuable technology requiring more human labor to produce, producing an overall increase in demand and higher average wages. When AGI creates new jobs, however, it can also do those new jobs. This could easily collapse the demand for human labor and drive average wages down to the cost of renting a robot and some compute (of course, there are also jobs that are valuable because a human is doing them, such as creating hand-made goods or producing art, and the Baumol effect could possibly allow for these to be very high-paying in a post-AGI world- though whether that narrow category of jobs can expand to provide full employment is anyone's guess).
Now, a fully or mostly post-labor economy could be a very good thing in principle- we could just redistribute the products of the robot farms and factories and everyone would be free to pursue their passions without needing to worry about money. But the incentives this creates are very worrying. Ordinary workers currently have a great deal of influence over governments and the owners of capital due to the value of their labor- governments and companies benefit financially by treating average people well. Democracy and social services increase tax revenue and selling products to consumers generates profit. In a post-AGI world, however, this would no longer hold true. The incentive would be for owners of capital (whether capitalists or government leaders claiming to represent the people) to use that capital mostly for their own grandiose projects and status competitions, and anything provided to ordinary people through redistribution would, for the first time, be pure charity. Maybe this world would be wealthy enough that even a tiny amount of generosity would be enough to provide everyone with a more comfortable life than the current labor market- but maybe not.
All of this, of course, assumes that the AGI is well-aligned with human values and that there are no ethical implications with us owning what will effectively be sentient minds. And, of course, that ASI doesn't happen.
Artificial superintelligence is hypothetical AI that would compare with human intelligence the way ours compares with the intelligence of wild animals- intelligence that's able to easily work with concepts that even the best human geniuses can't understand, and which might give it affordances providing as much advantage over us as language and tool use provides us over wild animals. Only, wheres we perceive time similarly to other animals, this intelligence would be able to think and act in the world far faster. There are good technical reasons to think that AI like this is possible, and no particular reason to think that AI progress should stop at the level of human experts.
We very much wouldn't be able to control ASI- if we or the AGIs that built it successfully aligned it to human values, it might build for us some strange utopia. If not, it would probably treat us like a wildlife habitat on a construction site, and we'd probably all die.
So, yeah- I'm worried about AI. There's no predicting where the progress will plateau, and while there are some very good possible futures, the risk- especially of rushing ahead as quickly as possible- is enormous.
Thanks for this great essay. As someone who in your words “rides the line”, it’s very difficult to have my feelings so far from how I believe I should act. At a gut level, I feel incredible excitement and honestly relief at the thought that there will never be normal times again. That there’s a chance that I won’t have to slowly watch everyone I love die before I do so myself, while never getting to see what happens or why in this incredible story we get to partake in. But intellectually, I believe that this is, on expectation, the most critical juncture in history. One from which for perhaps the first time, there will be no second chances if we get wrong. With the stakes so high, we are obligated to proceed with every possible caution. It would be an act of unforgivable selfishness to try and deliver our utopia a little sooner at the risk of never doing so at all. So I believe we desperately need to pause because there’s a good chance we are hurtling towards the brink of disaster. But each time things get less normal, my gut is so, so happy.
People need "normal times" to develop relationships and community, to raise children, to thrive. I agree this technology is disruptive, but if it's too disruptive the people who adopt it will not reproduce. We are already seeing the failure to thrive of people raised on social media, and AI is sure to accelerate that trend.
I hope to see increased political control of tech companies. The point of democracy is that people have a say in how a country is run. Technology like this needs to be under democratic control.
Re Millennial Prize math problems. Strikes me as very similar to the chess playing computers of the 80s and 90s that eventually topped the best human players. These are intellectual rule based domains that computers are particularly well suited for. It shouldn’t impress us too much when machines become adept at them. Such success doesn’t promise similar results in other areas.
I’m sure these AI tools will have some major impacts. But rapid progress in coding or math tells us very little about what they might do in things like medicine or engineering. It might well be very little.
I don't disagree with the thrust of your take here. We're in for weird times. But I would advise heavy caution with predictions. Weird things happen in weird times and we should have incredible humility about what's going to happen.
The people who have been right so far have made some astute observations but they're not oracles. They're human beings relying on very inexact and imperfect heuristics (which is why most of us were dismissive of them) that happen to have been mind-blowingly insightful so far. Going forward as we enter weird world we need to remember that there are no oracles and no certainties in the way things will go.
One particularly remarkable thing to me personally is that - at the same time we hit unfathomable levels of machine intelligence - the United States is in its longest ever streak of <=4.5% unemployment. Not only have the superintelligent machines not replaced anyone yet, they don't even correlate with joblessness. Speaking as a software dev and thus a guy whose day-to-day is drastically altered by AI, I'm not seeing any sign of disemployment. On the contrary, experienced SWE hiring is undergoing a boom in part because OpenAI and Anthropic (the actual frontier) are hiring up all the best talent from top companies.
AGI-ists have been astoundingly frighteningly correct about the intellectual power of this tech...and confusingly wrong about its (lack of) economic impact.
You can’t absolutely rule anything out, I guess, but this Dyson sphere stuff is just nonsense. You’d have to discover whole new physics to even begin to attempt it. Is there enough matter in the solar system to build even a ringworld? A swarm of mini spaceships would take decades at least to move any substantial amount of it into position for use. Etc etc.
Intelligence isn’t the barrier, it’s the basic physics of matter and energy. These are fantasies of magic not predictions to treat seriously.
If by "normal" you mean "exactly like before", then sure, but humans have a remarkable ability to quickly adapt to most anything. It'll be a new normal soon enough.
See all the “it’s just hype” people who are by now so used to AI systems writing effective code that they dismiss the HuggingFace attack by saying “that’s not a failure, it just did what they told it to do, and now they’re hyping it up like it’s dangerous”!
An old Chinese proverb says "Better to be a dog in times of tranquility than a human in times of chaos." I happen to disagree. Also, I wish you hadn't opened with a quote from that old Marxist instigator Antonio Gramsci..
Here are a few of the themes I think we should all focus on.
Meaning: How do we hold onto and reinforce the things that make life good, like family, friends, aspiration, and meaningful work (not necessarily a job in all cases), in the face of accelerating technological change? It seems like we took a big L on this one from social media and smartphones, and arguably we've been going downhill on this front for the last 50 years. Let's try not to double down on our mistakes. I love the abundance of capitalism, but it tends to crowd out anything you can't pay for. How do we make sure our sources of meaning don't get crowded out by AI-generated, ad-supported slop in all its forms? I think investing in the positive will work better than trying to ban technology.
Safety: How do we avoid losing control to AI? Right now, people are focused on whether it could eventually kill us. I think this is the wrong focus. We should be focused on making sure that we don't cede control to AI in the name of competition. We're seeing this pressure among the AI labs now. They want to slow down but feel like they can't given competition from other labs and countries. This same pressure will apply to governments (including in the military) and companies. What happens when someone tells the President we need to have AI control the weapons because otherwise we'll lose our military superiority? We need to manage and not give in to this competitive pressure. Once we delegate authority to AI, it almost doesn't matter if and when it could kill us, because we're no longer in the driver's seat. I do think AI safety is important as well, but we're spending too much time quibbling about viruses versus cyber versus robots. The underlying issues are control and alignment.
Prosperity: AI has the potential to bring unprecedented prosperity, and robotics is not far behind. But it will only happen if we let it. Governance bottlenecks like multi-year approval timelines for housing construction, grid improvements, and so on will result in most of that prosperity being unrealized. We urgently need pragmatic deregulation in a number of fields.
Adaptation: This applies to all of society, but specifically to Jack's focus on education. We need to prepare ourselves, and our kids, for a truly new world order. We may not be able to derive much meaning from our jobs any more, and we need to feel self-actualized even if work and other social structures we rely on are automated out from under us. How do we educate kids and make them successful members of society in our sci-fi future?
I personally have lived by the plaint of the folk song from the 30's: "I'm just hanging around to see what happens next.'
I opted out of working in nuclear physics in '58 while on a company-sponsored trip to The Huntsville AL space center. I'm 86 and still just hanging around. The cartoon is brilliant. Now I'm going to help a neighbor. Physics is a handy tool!
Yes, I agree with most of this - AND IT'S GOOD! (mostly). What you didn't mention in this post is the Industrial Revolution. When humans learned how to harness mechanical power on a large scale, it changed the entire face of civilization. The shift from agrarian to manufacturing economy was massive, with extensive population displacement from rural areas to cities. I think that's the way to look at things. Not Doomers or Singularity hypesters. I'm firmly in the camp of "AI Is Normal Technology" - but normal technology can have huge, absolutely enormous, effects on society.
You might want to take a look sometime into what the real, literal, Luddites were talking about (if you don't know already).
To be tedious, note we've been in the future for a while now with a worldwide interconnected network - something that was Science Fiction for a long time.
What should we do? I don't know. I personally regret missing the opportunity to get rich fast in bubble-money. There's always such an opportunity at the start of a technology change, but I think it's past that point now. Learning the details of AI is probably a good bet if you're young programmer type. It's going to be a crowded field, but also I think an expanding one for a while.
In general, I'd advocate that society make sure the social change problems are moderated by sharing the created wealth. But I know how purely aspiration that idea is.
Can you expand on the reasoning that leads you to conclude that "AI is normal technology"?
It seems to me that it is anything but. It is not only the first technology that promises to exceed human capabilities in the one area that humans have always surpassed every other life form on this planet, the one that made us capable of dominating all others from the largest to the smallest, but the first technology that is almost certainly going to be capable of iteratively improving itself, without any human involvement required.
Certainly opposing automation to preserve jobs or the social status quo is unethical- the very real human cost of job displacement is properly solved with a generous social safety net, not by robbing future generations of economic progress.
However, AI isn't just automation. These are minds; not yet as general as human minds, but if current benchmark trends continue for another two or three years, they will be. Introducing an entirely new kind of intelligent labor into civilization is something deeply unprecedented. Our familiar heuristics about the benefits of industrial automation and social progress aren't going to be reliable indicators of what to expect- to predict accurately, we have to look closely at the details.
First, let's assume away ASI- assume that AI progress plateaus around the level of human experts and goes no further (aside from a few narrow fields like coding and math). In this near future, AI is able to pilot cheap mass-manufactured robots and drones as easily as we move our bodies, and will be able to do effectively everything a human worker can, only far cheaper. If this sounds far-fetched, I encourage you to look up the benchmark trends in robotics, long-horizon task completion, sample efficiency and general reasoning- straightforward extrapolation has AI passing the human level at all of these in '28 or '29.
When historic automation replaced a job, the overall demand for human labor didn't decrease- the surplus from automation allows for new kinds of more valuable technology requiring more human labor to produce, producing an overall increase in demand and higher average wages. When AGI creates new jobs, however, it can also do those new jobs. This could easily collapse the demand for human labor and drive average wages down to the cost of renting a robot and some compute (of course, there are also jobs that are valuable because a human is doing them, such as creating hand-made goods or producing art, and the Baumol effect could possibly allow for these to be very high-paying in a post-AGI world- though whether that narrow category of jobs can expand to provide full employment is anyone's guess).
Now, a fully or mostly post-labor economy could be a very good thing in principle- we could just redistribute the products of the robot farms and factories and everyone would be free to pursue their passions without needing to worry about money. But the incentives this creates are very worrying. Ordinary workers currently have a great deal of influence over governments and the owners of capital due to the value of their labor- governments and companies benefit financially by treating average people well. Democracy and social services increase tax revenue and selling products to consumers generates profit. In a post-AGI world, however, this would no longer hold true. The incentive would be for owners of capital (whether capitalists or government leaders claiming to represent the people) to use that capital mostly for their own grandiose projects and status competitions, and anything provided to ordinary people through redistribution would, for the first time, be pure charity. Maybe this world would be wealthy enough that even a tiny amount of generosity would be enough to provide everyone with a more comfortable life than the current labor market- but maybe not.
All of this, of course, assumes that the AGI is well-aligned with human values and that there are no ethical implications with us owning what will effectively be sentient minds. And, of course, that ASI doesn't happen.
Artificial superintelligence is hypothetical AI that would compare with human intelligence the way ours compares with the intelligence of wild animals- intelligence that's able to easily work with concepts that even the best human geniuses can't understand, and which might give it affordances providing as much advantage over us as language and tool use provides us over wild animals. Only, wheres we perceive time similarly to other animals, this intelligence would be able to think and act in the world far faster. There are good technical reasons to think that AI like this is possible, and no particular reason to think that AI progress should stop at the level of human experts.
We very much wouldn't be able to control ASI- if we or the AGIs that built it successfully aligned it to human values, it might build for us some strange utopia. If not, it would probably treat us like a wildlife habitat on a construction site, and we'd probably all die.
So, yeah- I'm worried about AI. There's no predicting where the progress will plateau, and while there are some very good possible futures, the risk- especially of rushing ahead as quickly as possible- is enormous.
Thanks for this great essay. As someone who in your words “rides the line”, it’s very difficult to have my feelings so far from how I believe I should act. At a gut level, I feel incredible excitement and honestly relief at the thought that there will never be normal times again. That there’s a chance that I won’t have to slowly watch everyone I love die before I do so myself, while never getting to see what happens or why in this incredible story we get to partake in. But intellectually, I believe that this is, on expectation, the most critical juncture in history. One from which for perhaps the first time, there will be no second chances if we get wrong. With the stakes so high, we are obligated to proceed with every possible caution. It would be an act of unforgivable selfishness to try and deliver our utopia a little sooner at the risk of never doing so at all. So I believe we desperately need to pause because there’s a good chance we are hurtling towards the brink of disaster. But each time things get less normal, my gut is so, so happy.
Great content, but also just wanted to say that you are an excellent writer.
People need "normal times" to develop relationships and community, to raise children, to thrive. I agree this technology is disruptive, but if it's too disruptive the people who adopt it will not reproduce. We are already seeing the failure to thrive of people raised on social media, and AI is sure to accelerate that trend.
I hope to see increased political control of tech companies. The point of democracy is that people have a say in how a country is run. Technology like this needs to be under democratic control.
Re Millennial Prize math problems. Strikes me as very similar to the chess playing computers of the 80s and 90s that eventually topped the best human players. These are intellectual rule based domains that computers are particularly well suited for. It shouldn’t impress us too much when machines become adept at them. Such success doesn’t promise similar results in other areas.
I’m sure these AI tools will have some major impacts. But rapid progress in coding or math tells us very little about what they might do in things like medicine or engineering. It might well be very little.
I don't disagree with the thrust of your take here. We're in for weird times. But I would advise heavy caution with predictions. Weird things happen in weird times and we should have incredible humility about what's going to happen.
The people who have been right so far have made some astute observations but they're not oracles. They're human beings relying on very inexact and imperfect heuristics (which is why most of us were dismissive of them) that happen to have been mind-blowingly insightful so far. Going forward as we enter weird world we need to remember that there are no oracles and no certainties in the way things will go.
One particularly remarkable thing to me personally is that - at the same time we hit unfathomable levels of machine intelligence - the United States is in its longest ever streak of <=4.5% unemployment. Not only have the superintelligent machines not replaced anyone yet, they don't even correlate with joblessness. Speaking as a software dev and thus a guy whose day-to-day is drastically altered by AI, I'm not seeing any sign of disemployment. On the contrary, experienced SWE hiring is undergoing a boom in part because OpenAI and Anthropic (the actual frontier) are hiring up all the best talent from top companies.
AGI-ists have been astoundingly frighteningly correct about the intellectual power of this tech...and confusingly wrong about its (lack of) economic impact.
You can’t absolutely rule anything out, I guess, but this Dyson sphere stuff is just nonsense. You’d have to discover whole new physics to even begin to attempt it. Is there enough matter in the solar system to build even a ringworld? A swarm of mini spaceships would take decades at least to move any substantial amount of it into position for use. Etc etc.
Intelligence isn’t the barrier, it’s the basic physics of matter and energy. These are fantasies of magic not predictions to treat seriously.
If by "normal" you mean "exactly like before", then sure, but humans have a remarkable ability to quickly adapt to most anything. It'll be a new normal soon enough.
See all the “it’s just hype” people who are by now so used to AI systems writing effective code that they dismiss the HuggingFace attack by saying “that’s not a failure, it just did what they told it to do, and now they’re hyping it up like it’s dangerous”!
An old Chinese proverb says "Better to be a dog in times of tranquility than a human in times of chaos." I happen to disagree. Also, I wish you hadn't opened with a quote from that old Marxist instigator Antonio Gramsci..
Normality is heaven to an irritated organism. Adaptation requires excess energy and that's seen as pain.
Yes.
Here are a few of the themes I think we should all focus on.
Meaning: How do we hold onto and reinforce the things that make life good, like family, friends, aspiration, and meaningful work (not necessarily a job in all cases), in the face of accelerating technological change? It seems like we took a big L on this one from social media and smartphones, and arguably we've been going downhill on this front for the last 50 years. Let's try not to double down on our mistakes. I love the abundance of capitalism, but it tends to crowd out anything you can't pay for. How do we make sure our sources of meaning don't get crowded out by AI-generated, ad-supported slop in all its forms? I think investing in the positive will work better than trying to ban technology.
Safety: How do we avoid losing control to AI? Right now, people are focused on whether it could eventually kill us. I think this is the wrong focus. We should be focused on making sure that we don't cede control to AI in the name of competition. We're seeing this pressure among the AI labs now. They want to slow down but feel like they can't given competition from other labs and countries. This same pressure will apply to governments (including in the military) and companies. What happens when someone tells the President we need to have AI control the weapons because otherwise we'll lose our military superiority? We need to manage and not give in to this competitive pressure. Once we delegate authority to AI, it almost doesn't matter if and when it could kill us, because we're no longer in the driver's seat. I do think AI safety is important as well, but we're spending too much time quibbling about viruses versus cyber versus robots. The underlying issues are control and alignment.
Prosperity: AI has the potential to bring unprecedented prosperity, and robotics is not far behind. But it will only happen if we let it. Governance bottlenecks like multi-year approval timelines for housing construction, grid improvements, and so on will result in most of that prosperity being unrealized. We urgently need pragmatic deregulation in a number of fields.
Adaptation: This applies to all of society, but specifically to Jack's focus on education. We need to prepare ourselves, and our kids, for a truly new world order. We may not be able to derive much meaning from our jobs any more, and we need to feel self-actualized even if work and other social structures we rely on are automated out from under us. How do we educate kids and make them successful members of society in our sci-fi future?
I personally have lived by the plaint of the folk song from the 30's: "I'm just hanging around to see what happens next.'
I opted out of working in nuclear physics in '58 while on a company-sponsored trip to The Huntsville AL space center. I'm 86 and still just hanging around. The cartoon is brilliant. Now I'm going to help a neighbor. Physics is a handy tool!