Crap critiques of AI doomerism: AFR edition
As the AI doom debate has moved from a fringe topic into the mainstream, it has taken a predictable downswing in quality.
So in response to that downswing, let me take on this week’s contribution in the form of Richard Holden’s AFR opinion piece AI doomers are scaring us stupid (archived copy).
Holden provides a great mix of ad hominem attacks, non sequiturs and errors, so let’s work through it.
Holden’s basic case is that doomers are scaring people. The result is that we’re sceptical of AI when it is the best shot we have of delivering the productivity growth that underpins increasing living standards.
I can see a coherent argument shaped around this – Australia has one of the lowest levels of trust in AI in the world – but unfortunately, Holden doesn’t make it.
Holden traces the beginning of AI doomerism to Effective Altruism, taking us from Peter Singer’s drowning child thought experiment to Will MacAskill’s idea of earning to give (earn more, donate more). Along the way, he bizarrely cites Enron’s Jeff Skilling as an example of what can go wrong when people try to earn money – without mentioning Sam Bankman-Fried. He then claims that Effective Altruism “managed to mutate into an even more deadly strain”:
Effective altruists noted, innocuously and correctly, that if a choice or event had a negative expected value, then it should be avoided. For instance, nobody should agree to a bet on the flip of a coin that pays $1 if you choose correctly but costs you $1.05 if you choose incorrectly. Yup.
This led the EA crowd to the observation that a tiny probability of an event that leads to the end of humanity is to be avoided at all costs.
This in turn led effective altruists to worry about pandemics and AI. (Holden doesn’t actually mention pandemics. Why mention what they’ve got right in your rant …)1
Kind of a weird tour of existential concerns about AI. No mention of the people who were concerned about AI risks before Effective Altruism, such as Alan Turing, I.J. Good and Nick Bostrom. Rather than inventing these concerns, effective altruists took up an existing debate, influenced in part by Bostrom’s Superintelligence and Eliezer Yudkowsky’s advocacy.
Holden likens the focus on small probabilities to Edward Teller’s suggestion that the Trinity nuclear test might ignite the atmosphere (Holden incorrectly labels Teller as “Nobel Prize-winning”). Hans Bethe, who actually did win a Nobel, doubted the risk and challenged some of Teller’s assumptions. Teller and others worked through the calculations before the Trinity test to confirm Bethe’s instinctive reaction, although their detailed report wasn’t published until 1946. (Holden credits the calculations to Bethe, but the article he links to suggests the calculations were done by Teller, Emil Konopinski and Cloyd Marvin.)
What is notable here is that they didn’t just charge off and blow up the bomb when considering the risk. They investigated it. They revisited the same question for the first hydrogen bomb. It is also notable that Teller was part of the team working on the bomb. Not exactly a doomer trying to stop “progress” based on tiny probabilities.
What of Holden’s reference to “a tiny probability”?
A 2024 survey of AI researchers by Grace and friends found the following:
In this survey, researchers assigned an 18% chance on average of future AI advances causing human extinction or similarly permanent and severe disempowerment of the species, and most researchers assigned at least a 10% chance to such outcomes.
The median probability was 10%. Some put zero, but tiny probabilities?
What of the lower bar of catastrophic harm? Here’s one set of estimates from an MIT and University of Queensland Delphi study of 272 AI experts:
In a business-as-usual scenario, experts judged a more than 10% chance of catastrophic outcomes (i.e., ‘more than 1 million human deaths or more than a USD 100B in financial loss or civilizational-scale intangible impacts’) from 18 of our 24 AI risk domains over the next five years.
This hardly suggests concern about catastrophic AI harm rests on tiny probabilities.
Holden’s next leap is a grand one, as he turns to last week’s hearings of the Joint Select Committee on AI:
We heard a lot of speculation, backed up by the thinnest of evidence, about surveillance and machine-led micromanagement but very little about drug discovery, personalised medicine, and technology-enabled human flourishing.
Surveillance and micromanagement? Holden has gone through all this setup – the path from Effective Altruism to doomerism – to talk about concerns that effective altruists are often accused of neglecting. The “thinnest of evidence” included a survey of what employers are actually doing, worker polling and research on how AI-enabled surveillance is being used.
And there was plenty on health benefits, including appearances by the Australian Medical Association and the Royal Australian and New Zealand College of Radiologists. Even the Health Services Union representative noted the benefits. Many of them paired the benefits with safeguards (as you’d expect from any self-interested group), but the benefits were hardly ignored. They might have been even more ambitious – cancer cured by 2030! – but we’d probably need someone who is also concerned about existential risk to claim that. (Paging Dario Amodei …)
I’ve listened to many of those Committee hearings or read the transcripts. Last week’s discussions were largely about those mundane types of risks that doomerism supposedly takes attention from: jobs, copyright and worker surveillance.
Earlier hearings featured more “doomer-ish” organisations such as Good Ancestors, an Effective Altruism-adjacent organisation (that also supports frontier AI training in Australia), and Global Shield Australia, but that isn’t really Holden’s target. We see this in the following ad hominem paragraph:
The doomer industrial complex – often led by pseudo-academics who think getting to borrow the title “professor” makes them scholars and who hide behind terms like “human-centred” technology – should be seen for what it is: a bunch of folks with little expertise who make their nut from scaring the heck out of regular folks.
It isn’t too hard to see who Holden is targeting here with his references to borrowing the title of professor and hiding behind “human-centred”: UTS’s Human Technology Institute and its “Industry Professors”.2 HTI’s Ed Santow discussed the growth in tech surveillance of workers when he appeared before the Committee on 20 October. He also co-authored a Guardian piece (archived copy) that week describing “micromanagement by machine”, which is almost exactly Holden’s phrase.
But despite the underhanded jibe, it’s worth considering Holden’s note about “folks with little expertise” in more detail. In complaining about doomers, he ignores the masses of experts who have worries: Geoffrey Hinton (who, unlike Teller, actually has a Nobel Prize), lab leaders such as Dario Amodei, and academics such as Stuart Russell. Holden notes Jacob Coxon had “15 minutes of fame” without mentioning his years of experience in pre-training AI. Many experts and insiders are sounding the alarm. But why would Holden bother engaging with their arguments when he can tell some random story about these deluded effective altruists?
Let’s now return to Holden’s opening fact that he blames on the doomers:
… AI has a serious trust problem.
According to the Pew Research Centre, 49 per cent of Australians are “more concerned than excited” about the use of AI in daily life.
Only 13 per cent are “more excited than concerned”. On this measure, we’re the third most negative country about AI.
The report Holden derives this fact from doesn’t say why Australians are concerned. Is it because of the effective altruists? Is it because of HTI? Why is Australia lower than other countries?
Answering that question would take some actual work. Pew’s survey doesn’t contain any questions on why they’re concerned, but a survey by the University of Melbourne and KPMG carries some hints.
That survey tracks trust in AI and finds Australia to be down the bottom of the rankings for perceived trustworthiness. But for concerns about risks, Australia is broadly middle of the road, which doesn’t fit a story of doomers leading to abnormally low trust in Australia.
Where Australia seems weakest is in expectations or experience of AI’s benefits, which the authors note is associated with trust. Maybe someone needs to make a better case for the benefits, but don’t blame the effective altruists.
Footnotes
As a random aside, these drive-bys at Effective Altruism smell like the critiques of economics that start with stories of a secretive meeting in Switzerland in 1947.↩︎
Quick disclosure here – I’m employed by UTS and know some of the folks at HTI. I haven’t worked with them and, to be honest, would like to see them focus more on catastrophic risk. It was that familiarity that tipped me off as to who Holden was referring to.↩︎