I usually avoid shooting off my mouth on social media, but I’m a BIG fan of Tomas and this is one of the first times I think he’s way off base.
Everyone needs to take a breath. The AI apocalypse isn’t nigh!
Who am I? I’ve watched this movie from the beginning, not to mention participated in it. I got my PhD in AI in 1979 specializing in NLP, worked at Stanford then co-founded four Silicon Valley startups, two of which went public, in a 35-year career as a tech entrepreneur. I’ve invented several technologies, some involving AI, that you are likely using regularly if not everyday. I’ve published three award-winning or best-selling books, two on AI. Currently I teach “Social and Economic Impact of AI” in Stanford’s Computer Science Dept. (FYI Tomas’ analysis of the effects of automation – which is what AI really is – is hands down the best I’ve ever seen, and I am assigning it as reading in my course.)
May I add an even more shameless self-promotional note? My latest book, “Generative Artificial Intelligence: What Everyone Needs to Know” will be published by Oxford University Press in Feb, and is available for pre-order on Amazon: https://www.amazon.com/Generative-Artificial-Intelligence-Everyone-KnowRG/dp/0197773540. (If it’s not appropriate to post this link here, please let me know and I’ll be happy to remove.)
The concern about FOOM is way overblown. It has a long and undistinguished history in AI, the media, and (understandably so) in entertainment – which unfortunately Tomas sites in this post.
The root of this is a mystical, techno-religious idea that we are, as Elon Musk erroneously put it, “summoning the beast”. Every time there is an advance in AI, this school of thought (superintelligence, singularity, transhumanism, etc.) raises it head and gets way much more attention than it deserves. For a bit dated, but great deep-dive on this check out the religious-studies scholar Robert Geraci’s book “Apocalyptic AI: Visions of Heaven in Robotics, Artificial Intelligence, and Virtual Reality”.
AI is automation, pure and simple. It’s a tool that people can and will use to pursue their own goals, “good” or “bad”. It’s not going to suddenly wake up, realize it’s being exploited, inexplicably grow its own goals, take over the world, possibly wipe out humanity. We don’t need to worry about it drinking all our fine wine and marrying our children. These anthropomorphic fears are fantasy. It is based on a misunderstanding of “intelligence” (that it’s linear and unbounded), that “self-improvement” can runaway (as opposed to being asymptotic), that we’re dumb enough to build unsafe systems and hook them up to the means to cause a lot of damage (which, arguably, describes current self-driving cars). As someone who has built numerous tech products, I can assure you that it would take a Manhattan Project to build an AI system that can wipe out humanity, and I doubt it could succeed. Even so, we would have plenty of warning and numerous ways to mitigate the risks.
This is not to say that we can’t build dangerous tools, and I support sane efforts to monitor and regulate how and when AI is used, but the rest is pure fantasy. “They” are not coming for “us”, because there is no “they”. If AI does a lot of damage, that's on us, not "them".
There’s a ton to say about this, but just to pick one detail from the post, the idea that an AI system will somehow override it’s assigned goals is illogical. It would have to be designed to do this (not impossible…but if so that’s the assigned goal).
There are much greater real threats to worry about. For instance, that someone will use gene-splicing tech to make a highly lethal virus that runs rampant before we can stop it. Nuclear catastrophe. Climate change. All these things are verifiable risks, not a series of hypotheticals and hand-waving piled on top of each other. Tomas could write just as credible a post on aliens landing.
What’s new is that with Generative AI in general, and Large Language Models in particular, we’ve discovered something really important – that sufficiently detailed syntactic analysis can approximate semantics. LLMs are exquisitely sophisticated linguistic engines, and will have many, many valuable applications – hopefully mostly positive – that will improve human productivity, creativity, and science. It’s not “AGI” in the sense used in this post, and there’s a lot of reasonable analysis that it’s not on the path to this sort of superintelligence (see Gary Marcus here on Substack, for instance).
The recent upheaval at OpenAI isn’t some sort of struggle between evil corporations against righteous superheroes. It’s a predictable (and predicted!) consequence of poorly architected corporate governance, and inexperienced management and directors. I’ve had plenty of run-ins with Microsoft, but they aren’t going to unleash dangerous and liability-inducing products onto a hapless, innocent world. They are far better stewards of this technology than many nations. I expect this awkward kerfuffle to blow over quickly, especially because the key players aren't going anywhere, their just changing cubicles.
Focusing on AI as an existential threat risks drowning out the things we really need to pay attention to, like accelerating disinformation, algorithmic bias, so-called prompt hacking, etc. Unfortunately, it’s a lot easer to get attention screaming about the end of the world than calmly explaining that like every new technology, there are risks and benefits.
It’s great that we’re talking about all this, but for God sake please calm down! 😉
Super interesting. Thanks for your comment and kind words, Jerry, and congrats on your successes! I hope your book becomes a best seller!
I've also built tech products involving AI, including NLPs. In my case, that doesn't equip me to opine on the subject (but yours might!). Rather, I'd say what has convinced me that the doom scenario is possible (I said 20% probability) is having read a lot from Yudkowsky, Bostrom, and Tegmark, coming up with dozens of counters to the Doom scenario, and always realizing they had thought about it (especially Yudkowsky) and had a response to it. The ability to follow every step of the reasoning, and this track record of not being able to find holes in it, is what leads me to conclude the doom scenario is possible. In fact, I'd argue all my deductive reasoning ability tells me this is very likely, and I only reduce the odds inductively, based on previous cases, and through hope.
As such, I'm willing to engage in the nitty gritty of these debates, to figure out the holes in the logic, but other types of arguments like those of authority won't work with me (Yudkowsky has more than any of us!).
Onto the specifics:
• It would take a Manhattan project to wipe out humanity: What do you think OpenAI is? 750 of the most intelligent ppl in the field, working for years on this? Or DeepMind? These are very serious projects.
• AI is automation the same way as brains are automation. I find the comparison very apt. We know humans are conscious and believe they're intelligent, but we are not structurally different from apes. Based on what we know, we just have more of the same brain structure as apes. It turns out that scaling neural processing power gives you human brains. For me to be convinced that AGI is impossible, you'd have to explain to me what is the fundamental difference between an organic and a synthetic brain
• The speed of improvement is easy to explain: We're subject to evolution, which means improvements through random genetic variations and sexual reproduction. At this speed, it takes millions of years for cognitive improvements to have massive impact, and even then, it's all always limited by things like resources and sexual reproduction. Human brains are geared towards having sex and surviving, not having raw intelligence. None of these constraints are true in an inorganic brain.
• I am not saying that an AGI "inexplicably grows its own goals". Rather, that humans inadvertently or purposefully embed a goal when creating it (if it doesn't optimize for something it can't work), and this goal is not fully aligned with what humans want (what appears to be the case with all goals we can think of today). The goal can be as simple as gradient descent to predict the next best token (what LLMs do), or we might be able to force the 3 laws of robotics into an AI. These are just examples of such goals that, when scaled, are misaligned with what humans want ni general. So this is not as you say overriding human goals, but rather running with the ones we give it and turning out misaligned (because we don't know what we want).
• A virus that kills all humanity is also a full-on existential risk. Climate change and nuclear war are not, because in both cases the vast majority of civilization survives. Worst case scenario, enough humans survive to come back to where we are in a few thousand/hundred years. This is not the case for a virus or AI. They can wipe us out. Arguably, very few humans want to eliminate all humans, and most of those who do (all?) don't have the capacity to make it happen.
• I'm not arguing that LLMs are AGI, or on its path. I do believe deep learning is though, but I'm not married to the idea. Another infrastructure could turn out to work best. Here what matters is not the architecture, but rather: Are we close to AGI? If there's a chance it could be here in a matter of decades, then all alarms should go off.
• I'm not saying there are evil people here. They just optimize differently for somewhat different goals. The board seems very focused on safety, more so than Altman. If this is true, it is very important.
• None of the alternative issues you mention have a risk of wiping out humanity. Therefore, they are less important
Tomas (et. al.) – thanks for the thoughtful replies. In the spirit of closure, I’ll try to follow the convention of court filings that replies should be shorter than the original briefs. 😉
Overall reply: The risk of runaway AI leading to extinction isn’t zero – I certainly can’t guarantee that it won’t happen. It’s just very low, there’s plenty we can do about it, and there’s virtually no evidence that we’re anywhere close to it happening. It’s “fun” to think about, but we’ve got more important (and predictable) problems to worry about. Please tone down the inflammatory rhetoric!
I’m not as familiar with Yudkowsky and his work (shame on me), but I have debated Bostrom publicly and talked with Tegmark, and being more frank than I probably should be in writing, neither seems to be rooted in the realities and limitations of AI software engineering. The former is purely a philosopher, the latter a cosmologist. Both have deep roots in the “Future of Life/Transhumanism” movement, which I regard as a mostly spiritual (as opposed to scientific) endeavor. Smart and interesting guys, for sure, but don’t count on them to invent the light bulb or predict its practical impact.
I agree with you that its theoretically possible, indeed likely, that eventually we’ll write programs with capabilities that rival (or exceed) human brains in many or most ways, but it’s not clear what this means as a practical matter. (I can make a decent argument that we are already there.) But whether it’s fair to say they will have “minds”, are “conscious”, or have “free will”, and by implication deserve some sort of empathy or rights from us, is another matter entirely. I don’t think so. My headline on this is it can't happen unless they can "perceive" time. (I've had some mindblowing "conversations" with GPT-4 about its inability to do this, before OpenAI literally cut me off for violating their T&Cs - go figure! (Stochastic parrots my ass.) You’re probably aware there’s a longstanding raging debate about what these terms even mean, which has literally gotten nowhere. It’s rather funny that one camp of respected philosophers recently published a group paper accusing another camp of respected philosophers of being charlatans.
My point of view is that we don’t currently have a valid scientific framework for addressing this question. I cover all this in detail in my new book (another shameless plug!), where I first make a rock-solid argument that computers can’t be conscious, then another rock-solid argument that they absolutely can.
Even if AGI in the sense you mean is imminent, a prerequisite for existential risk is that for whatever reason, they are oblivious or actively hostile to human concerns, as Hollywood frequently depicts. This is the core flaw with the “existential risk” theories: On the one hand, AGIs are all-powerful and maniacally goal seeking, but on the other, they can’t be reasoned with, constrained by humans or other equally powerful programs, or take a more nuanced stance. Why the F knowingly wipe us out? To make paperclips? What sense does that make? Worst case, as I’ve argued in my book “Humans Need Not Apply”, they would want us around for biodiversity purposes.
The evidence that advanced AI can be “socialized” is already here. That’s how the LLM companies use RLHF (Reinforcement Learning from Human Feedback) to hone their products. I think it’s really interesting that constraints are not “built in”, they just explain what good robots should and shouldn’t do. I call this “finishing school” in my new book.
A little industry inside baseball: OpenAI is all the rage, but DeepMind has a much deeper bench, deeper pockets, a lot more serious stuff on the drawing boards, and more grounded management and goals IMO.
TV tip: For a frighteningly realistic yet hilarious take on AGI taking over, check out the series “Mrs. Davis”. Stick with it to the end where they explain the “paperclip”-like goal and how the program got loose and ran amok in the first place. Jerry says five stars.
Tomas, this sounds a bit silly but I’m serious: I would encourage you to do a similar deep dive on aliens landing/first contact. I’ve spent time with people at the SETI Institute (Search for Extraterrestrial Intelligence), and it’s really quite mysterious why this hasn’t happened yet. This “existential risk” is in the same category as runaway AI in my opinion. And like the AGI discussion, I think you will have a hard time poking any holes in their arguments.
Well I think I’ve hit my self-imposed word limit, hope this is food for thought!
Bostrom and Tegmark have been great at introducing me to great illustrations of these problems. Yudkowsky is the one that seems to have thought of everything.
Agreed that the consciousness & free will debate doesn't lead anywhere for me. I said what I had to say on the topic, and that's it.
Agreed AGI is not imminent, but it could be a matter of a handful of years, so attention is warranted, albeit not crazy urgency like with COVID.
Agreed that DeepMind needs a lot of scrutiny. Funnily, Yudkowsky is more concerned about Demis Hassabis than Altman I believe.
Mrs Davis looks good! I don't have Peacock tho...
Alien probability is much lower than AI!
The biggest issue with misalignment is instrumental convergence: No matter what it desires, odds are we will be on the way because we don't want to lose control and it would know it.
For me a risk of 1% of total human extinction is inacceptable.
Thank you Tomas and Jerry for this very interesting exchange.
Tomas- you got me really scared as I remember your Covid letter was a very useful wake-up call for me at the time.
I don’t have the knowledge to understand all the specifics but your arguments are compelling and it was quite balancing to read Jerry’s reply after 24hrs of mild queasy worry.
Obviously the drama that has happened at Openai was for me the best proof that what you’re saying is plausible. It seems the convincing explanation for this extreme behavior from Openai board people. we’re talking about very smart people (or so we hope) so we better hope they had good reason for acting like this.
Several things to
understand the board’s motives:
-If China is as engaged in this AI race as the Silicon Valley guys, what good would it be to humanity to damage one of the best contenders ? Was the board’s motive just to put the subject of alignment on the table and in the public’s eye?! (They did big time!)
-Why wouldn’t them have anticipated the backfiring of key stakeholders (employees, MS, Ilya…) or on the contrary, did they anticipate it and was it just a stunt instead of purely resigning (which is the current outcome).
Other food for thoughts that your article brought : is it only our brains that set us apart? Human Intelligence is also emotional and intuitive. This can’t easily be automated… or can it?
For my own peace of mind I choose to believe Jerry is right and Openai board is clumsy …but I nonetheless will be reading your further newsletters with an accrued interest…
CHINA: Helen Toner, one of the ousted board members at OpenAI, has actually some great points about this. She is an expert in Chinese AI, and says (1) their data is a shitshow, and (2) the fact that they need to censor it so much makes it go much more slowly and is less useful. She says China is a fast-follower, not a lead. One of the reasons to keep advancing fast in the West
ANTICIPATION: This doesn't happen! When has this happened? Jobs was ousted from Apple and nothing happened. CEOs are frequently ousted. They could have figured it out if they had played mental chess on the ramifications, but that's probably not what they paid attention to. They had enough to think about when considering AI safety
HUMAN BRAINS: I talk about it in today's article. The short is: a human brain is a physical thing that can be replicated. As long as it uses physics, it can be replicated. Emotions and intuitions are simply decisions that don't reach consciousness ("System 1 vs System 2") to save processing power. Totally modelable.
Without commenting on the wider discussion, I wanted to point out that at least certain people/teams within OpenAI think theres a good chance we’ll get to AGI to or beyond this decade: https://openai.com/blog/introducing-superalignment
Both the comment and response are valuable here... in the end, I concur with you Tomas that it's essentially "carbon chauvinism" to believe that only organic brains can reach the point of self-awareness/sentience. Silicon neural networks will almost certainly get there and, as you note, much more quickly than natural evolution allows.
It's very interesting to read viewpoints from intelligent people which suggest that climate change is not an existential threat. I believe the effective altruists also hold this view. I suspect this comes from the rather arrogant mindset which says that a) tech will come to the rescue or b) the pulverizing effect of mass extinctions, resource wars, rampant disease and unimaginable and constant weather events won't drive the human species back to the brink of survival. I wish I was as confident in our future in the face of the current evidence.
I hear you. I've made the case elsewhere. But yes, this is broadly what I'm saying. There's a difference between catastrophic and existential. Climate might be catastrophic, but not existential.
Catastrophic is enough to deserve lots of attention and work. But the attention and work should be proportional to the risk. Today, climate change receives more attention than AI safety. This must be inversed. Not by reducing focus on climate (we should increase it!) but by increasing the focus on AI even more.
It may just be catastrophic to us, living in the richest industrial societies in the world, but it will almost certainly be existential to hundreds of millions of people living along the equatorial region who will either perish or have to migrate to survive. But to where? And how will they feed themselves as they move? And what wars will threaten them as they seek safety and life? I'm not sure that 'existential' has to apply to the whole species to be truly horrific. How terrible does the loss of life have to be to qualify as 'existential'? 100 million, a billion, two, three, four...? Such jolly numbers to play with. :)
I'm using specific terminology here. Catastrophic and existential have specific meanings that I'll cover in the newsletter tomorrow.
That said, as I've mentioned in the past, it looks like climate change deaths might *shrink*, because most people in hot countries don't die of heat, but cold (unprepared), and vice-versa, and also because countries will have time to adapt.
Your lack of confidence (and Tomas’) are both unwarranted and rooted in the same psychological tendencies that have always led to doom thinking scenarios. These go back to the prehistory of humanity and can be seen in stories in mythology and religion to the theoretically logic based arguments of today. It is built into the human mind to think like Tomas has out you have about climate. Neither are any more right than the religious philosophers of the past. Neither are more likely predictive of the future than past doom scenarios.
My strongest suggestion is to focus on the more immediate problems that you can actually impact. The future will be fine.
I think both Climate change and funding ways to repair the damage done to our planet is vital for our survival, but also believe creating a safe AI protocol is also vital or our survival may not be ensured for long after solving our climate and planet destruction issues....which I hope AI can assist us in doing.
I think your argument lacks falsifiability. Just being able to imagine the adverse effects is not a sufficient argument that the adverse effects are likely to take place.
AI is one of the most disruptive and innovative technologies to emerge in the last ten years, and it’s seen a meteoric rise in just the past twelve months, but there is a lot of evidence to suggest we’re not even close to AGI, despite what your models show.
What’s largely missing in general AI models is context. Take your stock picking scenario, for example. The instant a general intelligence shorts a stock, the price will adjust to changing circumstances, because you have millions of other traders buying and selling and reacting to price fluctuations in real time. The idea that a general intelligence can manipulate the stock market is more of what the original poster describes as a mystical, techno-religious idea, because now you're in the business of constructing an entirely new reality, not merely acting with better intelligence (not to mention there are laws against insider trading).
Every intelligence is contextual within the context of the environment it's in. For all the advances in AI, they are still solving deterministic, closed-set, finite problems using large amounts of data. But living organisms that exist in nature do not operate at that surface level. We’re not even close to mapping the entirety of the brain (let alone modeling it), you can’t just abstract that out and call it AGI.
Just as well, the paperclip maximizer theory is an apt thought experiment for the existential risk posed by AI, but what exactly is testable about that hypothesis? What would falsify the hypothesis? It's the same thing with positing that because humans "don't know what they want," that AIs will develop runaway goals misaligned with the best interests of humans. It’s a good philosophical question, but it’s not a testable hypothesis.
I enjoy your takes, but I agree with the original poster here. I think a good amount of moral panic exists in what you're arguing.
Thanks! I think there might be a piece of info worth clarifying here.
Is the likelihood of this scenario less than 1% in the coming 10 years?
If you can resoundingly say "yes!" and defend it, then that would be convincing.
Otherwise, you should be freaking out.
That's the thing. I don't need to prove that this will happen. The burden is on the other side because of the consequences. YOU would need to prove that this CAN'T happen.
I don't even think there's a one percent chance it happens in our lifetime, much less the next ten years. But I'm gonna push back on the way you're framing this issue, you're thinking about the probability of an event based on what you perceive is the worst possible outcome (human extinction) and then working your way backwards. We can disagree on where the burden of proof lies, but I happen to think it rests on the people saying AGI will happen relatively soon. That's the position that needs to be defended, not its opposite. Because I have yet to see any hard, scientific evidence to suggest that we're anywhere close to achieving AGI.
I do think the burden of proof in *safety* lies in those saying it’s safe, but the burden of proof is on those who think it comes soon to prove it, so I agree with you on that.
Thanks. I think you have channeled my late brother for me. I, an artist. He an early computer scientist with amazing vision now he ha past some time ago. A very long time in computer terms. I have been wondering about the AI “discussion” and I kept thinking he’d say something as you did in essence if not specifics. He was always the smartest guy in the room. Miss him. Thanks for your time to answer here. Happy thanksgiving.
Nov 22, 2023·edited Nov 22, 2023Liked by Tomas Pueyo
Thanks, Thomas. That’s very sweet of you to say. And yes, anytime I get to share his memory is a good moment. That is AI enough. Happy Thanksgiving to you as well.
Tomas and friends:
I usually avoid shooting off my mouth on social media, but I’m a BIG fan of Tomas and this is one of the first times I think he’s way off base.
Everyone needs to take a breath. The AI apocalypse isn’t nigh!
Who am I? I’ve watched this movie from the beginning, not to mention participated in it. I got my PhD in AI in 1979 specializing in NLP, worked at Stanford then co-founded four Silicon Valley startups, two of which went public, in a 35-year career as a tech entrepreneur. I’ve invented several technologies, some involving AI, that you are likely using regularly if not everyday. I’ve published three award-winning or best-selling books, two on AI. Currently I teach “Social and Economic Impact of AI” in Stanford’s Computer Science Dept. (FYI Tomas’ analysis of the effects of automation – which is what AI really is – is hands down the best I’ve ever seen, and I am assigning it as reading in my course.)
May I add an even more shameless self-promotional note? My latest book, “Generative Artificial Intelligence: What Everyone Needs to Know” will be published by Oxford University Press in Feb, and is available for pre-order on Amazon: https://www.amazon.com/Generative-Artificial-Intelligence-Everyone-KnowRG/dp/0197773540. (If it’s not appropriate to post this link here, please let me know and I’ll be happy to remove.)
The concern about FOOM is way overblown. It has a long and undistinguished history in AI, the media, and (understandably so) in entertainment – which unfortunately Tomas sites in this post.
The root of this is a mystical, techno-religious idea that we are, as Elon Musk erroneously put it, “summoning the beast”. Every time there is an advance in AI, this school of thought (superintelligence, singularity, transhumanism, etc.) raises it head and gets way much more attention than it deserves. For a bit dated, but great deep-dive on this check out the religious-studies scholar Robert Geraci’s book “Apocalyptic AI: Visions of Heaven in Robotics, Artificial Intelligence, and Virtual Reality”.
AI is automation, pure and simple. It’s a tool that people can and will use to pursue their own goals, “good” or “bad”. It’s not going to suddenly wake up, realize it’s being exploited, inexplicably grow its own goals, take over the world, possibly wipe out humanity. We don’t need to worry about it drinking all our fine wine and marrying our children. These anthropomorphic fears are fantasy. It is based on a misunderstanding of “intelligence” (that it’s linear and unbounded), that “self-improvement” can runaway (as opposed to being asymptotic), that we’re dumb enough to build unsafe systems and hook them up to the means to cause a lot of damage (which, arguably, describes current self-driving cars). As someone who has built numerous tech products, I can assure you that it would take a Manhattan Project to build an AI system that can wipe out humanity, and I doubt it could succeed. Even so, we would have plenty of warning and numerous ways to mitigate the risks.
This is not to say that we can’t build dangerous tools, and I support sane efforts to monitor and regulate how and when AI is used, but the rest is pure fantasy. “They” are not coming for “us”, because there is no “they”. If AI does a lot of damage, that's on us, not "them".
There’s a ton to say about this, but just to pick one detail from the post, the idea that an AI system will somehow override it’s assigned goals is illogical. It would have to be designed to do this (not impossible…but if so that’s the assigned goal).
There are much greater real threats to worry about. For instance, that someone will use gene-splicing tech to make a highly lethal virus that runs rampant before we can stop it. Nuclear catastrophe. Climate change. All these things are verifiable risks, not a series of hypotheticals and hand-waving piled on top of each other. Tomas could write just as credible a post on aliens landing.
What’s new is that with Generative AI in general, and Large Language Models in particular, we’ve discovered something really important – that sufficiently detailed syntactic analysis can approximate semantics. LLMs are exquisitely sophisticated linguistic engines, and will have many, many valuable applications – hopefully mostly positive – that will improve human productivity, creativity, and science. It’s not “AGI” in the sense used in this post, and there’s a lot of reasonable analysis that it’s not on the path to this sort of superintelligence (see Gary Marcus here on Substack, for instance).
The recent upheaval at OpenAI isn’t some sort of struggle between evil corporations against righteous superheroes. It’s a predictable (and predicted!) consequence of poorly architected corporate governance, and inexperienced management and directors. I’ve had plenty of run-ins with Microsoft, but they aren’t going to unleash dangerous and liability-inducing products onto a hapless, innocent world. They are far better stewards of this technology than many nations. I expect this awkward kerfuffle to blow over quickly, especially because the key players aren't going anywhere, their just changing cubicles.
Focusing on AI as an existential threat risks drowning out the things we really need to pay attention to, like accelerating disinformation, algorithmic bias, so-called prompt hacking, etc. Unfortunately, it’s a lot easer to get attention screaming about the end of the world than calmly explaining that like every new technology, there are risks and benefits.
It’s great that we’re talking about all this, but for God sake please calm down! 😉
Super interesting. Thanks for your comment and kind words, Jerry, and congrats on your successes! I hope your book becomes a best seller!
I've also built tech products involving AI, including NLPs. In my case, that doesn't equip me to opine on the subject (but yours might!). Rather, I'd say what has convinced me that the doom scenario is possible (I said 20% probability) is having read a lot from Yudkowsky, Bostrom, and Tegmark, coming up with dozens of counters to the Doom scenario, and always realizing they had thought about it (especially Yudkowsky) and had a response to it. The ability to follow every step of the reasoning, and this track record of not being able to find holes in it, is what leads me to conclude the doom scenario is possible. In fact, I'd argue all my deductive reasoning ability tells me this is very likely, and I only reduce the odds inductively, based on previous cases, and through hope.
As such, I'm willing to engage in the nitty gritty of these debates, to figure out the holes in the logic, but other types of arguments like those of authority won't work with me (Yudkowsky has more than any of us!).
Onto the specifics:
• It would take a Manhattan project to wipe out humanity: What do you think OpenAI is? 750 of the most intelligent ppl in the field, working for years on this? Or DeepMind? These are very serious projects.
• AI is automation the same way as brains are automation. I find the comparison very apt. We know humans are conscious and believe they're intelligent, but we are not structurally different from apes. Based on what we know, we just have more of the same brain structure as apes. It turns out that scaling neural processing power gives you human brains. For me to be convinced that AGI is impossible, you'd have to explain to me what is the fundamental difference between an organic and a synthetic brain
• The speed of improvement is easy to explain: We're subject to evolution, which means improvements through random genetic variations and sexual reproduction. At this speed, it takes millions of years for cognitive improvements to have massive impact, and even then, it's all always limited by things like resources and sexual reproduction. Human brains are geared towards having sex and surviving, not having raw intelligence. None of these constraints are true in an inorganic brain.
• I am not saying that an AGI "inexplicably grows its own goals". Rather, that humans inadvertently or purposefully embed a goal when creating it (if it doesn't optimize for something it can't work), and this goal is not fully aligned with what humans want (what appears to be the case with all goals we can think of today). The goal can be as simple as gradient descent to predict the next best token (what LLMs do), or we might be able to force the 3 laws of robotics into an AI. These are just examples of such goals that, when scaled, are misaligned with what humans want ni general. So this is not as you say overriding human goals, but rather running with the ones we give it and turning out misaligned (because we don't know what we want).
• A virus that kills all humanity is also a full-on existential risk. Climate change and nuclear war are not, because in both cases the vast majority of civilization survives. Worst case scenario, enough humans survive to come back to where we are in a few thousand/hundred years. This is not the case for a virus or AI. They can wipe us out. Arguably, very few humans want to eliminate all humans, and most of those who do (all?) don't have the capacity to make it happen.
• I'm not arguing that LLMs are AGI, or on its path. I do believe deep learning is though, but I'm not married to the idea. Another infrastructure could turn out to work best. Here what matters is not the architecture, but rather: Are we close to AGI? If there's a chance it could be here in a matter of decades, then all alarms should go off.
• I'm not saying there are evil people here. They just optimize differently for somewhat different goals. The board seems very focused on safety, more so than Altman. If this is true, it is very important.
• None of the alternative issues you mention have a risk of wiping out humanity. Therefore, they are less important
Very important debate! Thanks for engaging
Tomas (et. al.) – thanks for the thoughtful replies. In the spirit of closure, I’ll try to follow the convention of court filings that replies should be shorter than the original briefs. 😉
Overall reply: The risk of runaway AI leading to extinction isn’t zero – I certainly can’t guarantee that it won’t happen. It’s just very low, there’s plenty we can do about it, and there’s virtually no evidence that we’re anywhere close to it happening. It’s “fun” to think about, but we’ve got more important (and predictable) problems to worry about. Please tone down the inflammatory rhetoric!
I’m not as familiar with Yudkowsky and his work (shame on me), but I have debated Bostrom publicly and talked with Tegmark, and being more frank than I probably should be in writing, neither seems to be rooted in the realities and limitations of AI software engineering. The former is purely a philosopher, the latter a cosmologist. Both have deep roots in the “Future of Life/Transhumanism” movement, which I regard as a mostly spiritual (as opposed to scientific) endeavor. Smart and interesting guys, for sure, but don’t count on them to invent the light bulb or predict its practical impact.
I agree with you that its theoretically possible, indeed likely, that eventually we’ll write programs with capabilities that rival (or exceed) human brains in many or most ways, but it’s not clear what this means as a practical matter. (I can make a decent argument that we are already there.) But whether it’s fair to say they will have “minds”, are “conscious”, or have “free will”, and by implication deserve some sort of empathy or rights from us, is another matter entirely. I don’t think so. My headline on this is it can't happen unless they can "perceive" time. (I've had some mindblowing "conversations" with GPT-4 about its inability to do this, before OpenAI literally cut me off for violating their T&Cs - go figure! (Stochastic parrots my ass.) You’re probably aware there’s a longstanding raging debate about what these terms even mean, which has literally gotten nowhere. It’s rather funny that one camp of respected philosophers recently published a group paper accusing another camp of respected philosophers of being charlatans.
My point of view is that we don’t currently have a valid scientific framework for addressing this question. I cover all this in detail in my new book (another shameless plug!), where I first make a rock-solid argument that computers can’t be conscious, then another rock-solid argument that they absolutely can.
Even if AGI in the sense you mean is imminent, a prerequisite for existential risk is that for whatever reason, they are oblivious or actively hostile to human concerns, as Hollywood frequently depicts. This is the core flaw with the “existential risk” theories: On the one hand, AGIs are all-powerful and maniacally goal seeking, but on the other, they can’t be reasoned with, constrained by humans or other equally powerful programs, or take a more nuanced stance. Why the F knowingly wipe us out? To make paperclips? What sense does that make? Worst case, as I’ve argued in my book “Humans Need Not Apply”, they would want us around for biodiversity purposes.
The evidence that advanced AI can be “socialized” is already here. That’s how the LLM companies use RLHF (Reinforcement Learning from Human Feedback) to hone their products. I think it’s really interesting that constraints are not “built in”, they just explain what good robots should and shouldn’t do. I call this “finishing school” in my new book.
A little industry inside baseball: OpenAI is all the rage, but DeepMind has a much deeper bench, deeper pockets, a lot more serious stuff on the drawing boards, and more grounded management and goals IMO.
TV tip: For a frighteningly realistic yet hilarious take on AGI taking over, check out the series “Mrs. Davis”. Stick with it to the end where they explain the “paperclip”-like goal and how the program got loose and ran amok in the first place. Jerry says five stars.
Tomas, this sounds a bit silly but I’m serious: I would encourage you to do a similar deep dive on aliens landing/first contact. I’ve spent time with people at the SETI Institute (Search for Extraterrestrial Intelligence), and it’s really quite mysterious why this hasn’t happened yet. This “existential risk” is in the same category as runaway AI in my opinion. And like the AGI discussion, I think you will have a hard time poking any holes in their arguments.
Well I think I’ve hit my self-imposed word limit, hope this is food for thought!
Jerry
Thank you, that's fair!
Bostrom and Tegmark have been great at introducing me to great illustrations of these problems. Yudkowsky is the one that seems to have thought of everything.
Agreed that the consciousness & free will debate doesn't lead anywhere for me. I said what I had to say on the topic, and that's it.
https://unchartedterritories.tomaspueyo.com/p/does-free-will-exist
Agreed AGI is not imminent, but it could be a matter of a handful of years, so attention is warranted, albeit not crazy urgency like with COVID.
Agreed that DeepMind needs a lot of scrutiny. Funnily, Yudkowsky is more concerned about Demis Hassabis than Altman I believe.
Mrs Davis looks good! I don't have Peacock tho...
Alien probability is much lower than AI!
The biggest issue with misalignment is instrumental convergence: No matter what it desires, odds are we will be on the way because we don't want to lose control and it would know it.
For me a risk of 1% of total human extinction is inacceptable.
Thank you Tomas and Jerry for this very interesting exchange.
Tomas- you got me really scared as I remember your Covid letter was a very useful wake-up call for me at the time.
I don’t have the knowledge to understand all the specifics but your arguments are compelling and it was quite balancing to read Jerry’s reply after 24hrs of mild queasy worry.
Obviously the drama that has happened at Openai was for me the best proof that what you’re saying is plausible. It seems the convincing explanation for this extreme behavior from Openai board people. we’re talking about very smart people (or so we hope) so we better hope they had good reason for acting like this.
Several things to
understand the board’s motives:
-If China is as engaged in this AI race as the Silicon Valley guys, what good would it be to humanity to damage one of the best contenders ? Was the board’s motive just to put the subject of alignment on the table and in the public’s eye?! (They did big time!)
-Why wouldn’t them have anticipated the backfiring of key stakeholders (employees, MS, Ilya…) or on the contrary, did they anticipate it and was it just a stunt instead of purely resigning (which is the current outcome).
Other food for thoughts that your article brought : is it only our brains that set us apart? Human Intelligence is also emotional and intuitive. This can’t easily be automated… or can it?
For my own peace of mind I choose to believe Jerry is right and Openai board is clumsy …but I nonetheless will be reading your further newsletters with an accrued interest…
Salut Marie! Gros plaisir de te revoir par ici!
CHINA: Helen Toner, one of the ousted board members at OpenAI, has actually some great points about this. She is an expert in Chinese AI, and says (1) their data is a shitshow, and (2) the fact that they need to censor it so much makes it go much more slowly and is less useful. She says China is a fast-follower, not a lead. One of the reasons to keep advancing fast in the West
ANTICIPATION: This doesn't happen! When has this happened? Jobs was ousted from Apple and nothing happened. CEOs are frequently ousted. They could have figured it out if they had played mental chess on the ramifications, but that's probably not what they paid attention to. They had enough to think about when considering AI safety
HUMAN BRAINS: I talk about it in today's article. The short is: a human brain is a physical thing that can be replicated. As long as it uses physics, it can be replicated. Emotions and intuitions are simply decisions that don't reach consciousness ("System 1 vs System 2") to save processing power. Totally modelable.
Looking forward to your opinions!
Without commenting on the wider discussion, I wanted to point out that at least certain people/teams within OpenAI think theres a good chance we’ll get to AGI to or beyond this decade: https://openai.com/blog/introducing-superalignment
Nice ideias.
I don´t know if the word has the exact meaning, but seams that alien landing is like a 'taboo' for some serious scientist.
Both the comment and response are valuable here... in the end, I concur with you Tomas that it's essentially "carbon chauvinism" to believe that only organic brains can reach the point of self-awareness/sentience. Silicon neural networks will almost certainly get there and, as you note, much more quickly than natural evolution allows.
It's very interesting to read viewpoints from intelligent people which suggest that climate change is not an existential threat. I believe the effective altruists also hold this view. I suspect this comes from the rather arrogant mindset which says that a) tech will come to the rescue or b) the pulverizing effect of mass extinctions, resource wars, rampant disease and unimaginable and constant weather events won't drive the human species back to the brink of survival. I wish I was as confident in our future in the face of the current evidence.
I hear you. I've made the case elsewhere. But yes, this is broadly what I'm saying. There's a difference between catastrophic and existential. Climate might be catastrophic, but not existential.
Catastrophic is enough to deserve lots of attention and work. But the attention and work should be proportional to the risk. Today, climate change receives more attention than AI safety. This must be inversed. Not by reducing focus on climate (we should increase it!) but by increasing the focus on AI even more.
It may just be catastrophic to us, living in the richest industrial societies in the world, but it will almost certainly be existential to hundreds of millions of people living along the equatorial region who will either perish or have to migrate to survive. But to where? And how will they feed themselves as they move? And what wars will threaten them as they seek safety and life? I'm not sure that 'existential' has to apply to the whole species to be truly horrific. How terrible does the loss of life have to be to qualify as 'existential'? 100 million, a billion, two, three, four...? Such jolly numbers to play with. :)
I'm using specific terminology here. Catastrophic and existential have specific meanings that I'll cover in the newsletter tomorrow.
That said, as I've mentioned in the past, it looks like climate change deaths might *shrink*, because most people in hot countries don't die of heat, but cold (unprepared), and vice-versa, and also because countries will have time to adapt.
Your lack of confidence (and Tomas’) are both unwarranted and rooted in the same psychological tendencies that have always led to doom thinking scenarios. These go back to the prehistory of humanity and can be seen in stories in mythology and religion to the theoretically logic based arguments of today. It is built into the human mind to think like Tomas has out you have about climate. Neither are any more right than the religious philosophers of the past. Neither are more likely predictive of the future than past doom scenarios.
My strongest suggestion is to focus on the more immediate problems that you can actually impact. The future will be fine.
I think both Climate change and funding ways to repair the damage done to our planet is vital for our survival, but also believe creating a safe AI protocol is also vital or our survival may not be ensured for long after solving our climate and planet destruction issues....which I hope AI can assist us in doing.
I think your argument lacks falsifiability. Just being able to imagine the adverse effects is not a sufficient argument that the adverse effects are likely to take place.
AI is one of the most disruptive and innovative technologies to emerge in the last ten years, and it’s seen a meteoric rise in just the past twelve months, but there is a lot of evidence to suggest we’re not even close to AGI, despite what your models show.
What’s largely missing in general AI models is context. Take your stock picking scenario, for example. The instant a general intelligence shorts a stock, the price will adjust to changing circumstances, because you have millions of other traders buying and selling and reacting to price fluctuations in real time. The idea that a general intelligence can manipulate the stock market is more of what the original poster describes as a mystical, techno-religious idea, because now you're in the business of constructing an entirely new reality, not merely acting with better intelligence (not to mention there are laws against insider trading).
Every intelligence is contextual within the context of the environment it's in. For all the advances in AI, they are still solving deterministic, closed-set, finite problems using large amounts of data. But living organisms that exist in nature do not operate at that surface level. We’re not even close to mapping the entirety of the brain (let alone modeling it), you can’t just abstract that out and call it AGI.
Just as well, the paperclip maximizer theory is an apt thought experiment for the existential risk posed by AI, but what exactly is testable about that hypothesis? What would falsify the hypothesis? It's the same thing with positing that because humans "don't know what they want," that AIs will develop runaway goals misaligned with the best interests of humans. It’s a good philosophical question, but it’s not a testable hypothesis.
I enjoy your takes, but I agree with the original poster here. I think a good amount of moral panic exists in what you're arguing.
Thanks! I think there might be a piece of info worth clarifying here.
Is the likelihood of this scenario less than 1% in the coming 10 years?
If you can resoundingly say "yes!" and defend it, then that would be convincing.
Otherwise, you should be freaking out.
That's the thing. I don't need to prove that this will happen. The burden is on the other side because of the consequences. YOU would need to prove that this CAN'T happen.
I don't even think there's a one percent chance it happens in our lifetime, much less the next ten years. But I'm gonna push back on the way you're framing this issue, you're thinking about the probability of an event based on what you perceive is the worst possible outcome (human extinction) and then working your way backwards. We can disagree on where the burden of proof lies, but I happen to think it rests on the people saying AGI will happen relatively soon. That's the position that needs to be defended, not its opposite. Because I have yet to see any hard, scientific evidence to suggest that we're anywhere close to achieving AGI.
Ok this is useful.
I do think the burden of proof in *safety* lies in those saying it’s safe, but the burden of proof is on those who think it comes soon to prove it, so I agree with you on that.
I wrote an article on this:
https://unchartedterritories.tomaspueyo.com/p/how-fast-will-ai-automation-arrive
Thanks Thomas, I'll read that. And I agree that there's dangers that need to be taken seriously, I just think they more solvable than people realize.
Thanks. I think you have channeled my late brother for me. I, an artist. He an early computer scientist with amazing vision now he ha past some time ago. A very long time in computer terms. I have been wondering about the AI “discussion” and I kept thinking he’d say something as you did in essence if not specifics. He was always the smartest guy in the room. Miss him. Thanks for your time to answer here. Happy thanksgiving.
Sorry for your loss. Glad I could make him come a bit alive. It might be that AI in the future makes him come even more alive. Happy thanksgiving!
Thanks, Thomas. That’s very sweet of you to say. And yes, anytime I get to share his memory is a good moment. That is AI enough. Happy Thanksgiving to you as well.