31/03/2016

Been reading, Q1 2016



I'm back on part-time university maths, which saps most of my real reading time. There is maybe something pathological about how irritable and small I feel when not reading in quantity. But half of life is about steering one's pathologies into productive rhythm, so whatever.


1/5: No.   4/5: 4/5: Very good.
2/5: Meh.   4*/5: Amazing but one read will do..
3/5: Skimmable.   5?/5: A possible 5/5.
3*/5: Mind candy.   5/5: Encore. A life companion.

JAN
  • Rogues, Villains and Eccentrics (2002) by William Donaldson. Addicting, horrible and hilarious biographies of British folly, banality and sin. A thousand years of tabloid gossip and popular madness, events too ephemeral for most serious historians: degradation, unchecked insanity and petty cruelty. But incredibly funny. The biographies are spaced out by Donaldson's wonderful little hooks, dry sentences that lead one on a wiki-walk:
    • ears, bagfuls of drying
    • universes, privileged to be part of a team working in many
    • drinking 'brain damage' while composing a speech for Michael Heseltine
    • coal merchants, remarkable
    • voluptuous Tartars and tun-bellied Chinese
    • dog on a diet of cats, feeding one's 12-stone
    • soft heart and 83 previous convictions, a
    He has particular obsessions, and the book is organised around this: the fate of gays throughout British history; criminal priests, eccentric spinster aristocrats, the line of succession of London ganglords from Jonathan Wild onward; politicians doing what they ought not; the odd fates private schoolboys often find themselves in... Obviously this is no demerit in an unsystematic historian. The modern gang biographies attest to his personal acquaintance with the big diamond geezers (which makes him a "silly bollocks", a foolish gang dilettante). His wit's mostly very dry, on occasion boiling over into outrage:
    Dodd's execution took place at Tyburn on 27 June 1777 and the outcry it occasioned has been recognized by some historians as a key moment in focusing public attention on the brutality of capital punishment. It seems more likely, however, that it was caused less by any broad change in public opinion than by the fact Dodd was of the same class as those protesting his execution. A 15-year-old orphan, John Harris, hanged on the same day for stealing two and half guineas, received no such support, least of all from Dr Johnson.

    Under the Misuse of Drugs Act 1971, the law was changed to ensure that the production and supply of dangerous drugs should henceforth be in the hands of criminal organisations. Some people have argued that this is not an ideal arrangement.
    I made the mistake of trying to read it over one week - so the endless succession of 18th century rapist officers being instantly pardoned and/or their victims being arrested kind of ran together. It is actually the best bog book ever and wants 4 slow months. I understand Britain a lot better now. The author would emphatically deserve an entry of his own in any future edition: astonishing wit, astonishing connections, astonishing potential, with little to show for it but a barrel of laughs and this.
    4*/5.



  • This Changes Everything: Capitalism vs the Climate (2014) by Naomi Klein. Thoughtful and exciting but not persuasive. As economics this is shaky, and as politics unlikely, but she remains one of the best journalists I know of. (i.e. person who works at the “These terrible and wonderful and unknown things are happening; here is what the people involved say. What might it mean?” level.) Considerable. Full review forthcoming.
    3*/5.



  • Selected Poems (1975-2011) by Jaan Kaplinski. A very broad swathe from Estonia's most stately rustic. He keeps a high eyrie but has a fatherly musk as well. It's a chilly nest though - occasionally anti-human:
    It gets cold in the evening. The sky clears.
    The wind dies out, and the smoke
    rises straight up. The flowering maple
    no longer buzzes. A carp
    plops in the pond. An owl hoots twice
    in its nest in the ash tree.
    The children are asleep. On the stairs,
    a long row of shoes and rubber boots.
    It happened near Viljandi: an imbecile boy
    poured gasoline on the neighbour's three-year-old
    and set him on fire. I ran for milk.
    You could see the yellow maple from far off
    between the birches and the spruce. The evening star
    was shining above the storehouse. The boy survived,
    probably maimed for life. The night will bring frost.
    Plentiful dew.
    He gets called a particularly European (a particularly Unionised) poet, and this is true enough: Kant's rationalist cool and Smith's pragmatism really are pedal notes in him. But there are snippets of nine languages in this mid-sized selection, including Sanskrit and Japanese (the ukiyo-e/mono-no-aware rhythms of which he owes a great deal to) and a poke of originals in pragmatic, wriggling English. That is, he's really a globalist. His own Estonian ("serious, greyish") is of course not remotely Indo-European, instead fluting and crashing, riverine, out of the Urals. (It would be silly to say that his work's bleak because some people he is descended from came from Siberia, but if I were a marketer rather than a gadfly it would be a good hook.) Let's complicate matters with two other sides, the paternal domestic and the wide-eyed enquirer:
    Lines do not perhaps exist; there are only points.
    Just as there are no constellations, only stars
    which we combine into water-carriers, fish, rams,
    virgins, scorpions and ourselves…
    Constellations, contours, profiles,
    outlines, ground plans, principles, reasons,
    ulterior motives and consequences…
    A solitary birch holds onto its last leaves by the woodshed.
    Or the leaves hold onto the birch.
    Or there is someone holds onto both,
    a child holding his father's and mother's hands at once.
    I am sorry for them – the child, the leaves
    the father, the birch and the mother.
    But I do not know, really, for whom: if the birch exists,
    if there are only points. I do not want the winter.
    But I do not know whether the winter really exists. There are only points.
    There are only molecules and atoms, which increasingly slowly,
    which is roughly the same as saying: warms disperses
    throughout space. Both the child's hands were cold.
    Night is coming - light is roughly the same as warmth.
    Light scatters in the empty room. New thoughts
    come so seldom. Your hand is warm. So is the night.
    The poem is ready. If the poem exists at all:
    there are only points. It is dark.
    This wonderful latter aside (and anti-poetry though he is) I do not like him constantly bringing up poetry; the poems where he does are often po-faced and contentless. But he is a master and it's his business what he chooses to cool by just gazing at it.
    4/5.



FEB

  • Accelerando (2004) by Charles Stross. His grandest statement so far: a scary family-dynasty epic told at that point in history where generational gaps grow unbridgeably vast on the spume of telescoping technological progression. First book is a wonderful freewheel through the near-future, with his technolibertarian booster protagonist – Sam Altman meets Richard Stallman meets Ventakesh Rao – running around as midwife to the future. Includes a nepotistic jaunt through Edinburgh because why not (it's a tech town after all). It is both funny and prescient (about e.g. our dependence on feeds and open-source expansion).
    Welcome to the early twenty-first century, human.
    It’s night in Milton Keynes, sunrise in Hong Kong. Moore’s Law rolls inexorably on, dragging humanity toward the uncertain future. The planets of the solar system have a combined mass of approximately 2 x 1027 kilograms. Around the world, laboring women produce forty-five thousand babies a day, representing 1023 MIPS of processing power. Also around the world, fab lines casually churn out thirty million microprocessors a day, representing 1023 MIPS. In another ten months, most of the MIPS being added to the solar system will be machine-hosted for the first time.
    The later books work less well; they become less and less convincing as we reach the singularity (his grasp of the physics and the economics of computers and space is characteristically excellent, and it's all hard enough) - more and more of that omniscient voiceover guy is needed.
    Not everyone is concerned with the deep future. But it’s important! If we live or die, that doesn’t matter—that’s not the big picture. The big question is whether information originating in our light cone is preserved, or whether we’re stuck in a lossy medium where our very existence counts for nothing. It’s downright embarrassing to be a member of a species with such a profound lack of curiosity about its own future, especially when it affects us all personally!
    I agree with Kahneman, though, that it is wrong to put as much weight on a weak ending as people tend to; the experiencing self, who was deeply impressed most of the time, should not be relegated so.
    In the distance, the cat hears the sound of lobster minds singing in the void, a distant feed streaming from their cometary home as it drifts silently out through the asteroid belt, en route to a chilly encounter beyond Neptune. The lobsters sing of alienation and obsolescence, of intelligence too slow and tenuous to support the vicious pace of change that has sandblasted the human world until all the edges people cling to are jagged and brittle.
    As always, many incredible thoughts embodied in very vivid scenes – it deserves the technical glossary supplied by fans here - and you've no regrets about spending time with him. But again I've the patronising sense that he fluffed it.

    Book I 5/5, Book II 3/5, Book III 2/5
    = 3*/5. [Free! here.]



  • Stamboul Train (1932) by Graham Greene. Better known as Orient Express. He tried to write a stupid book – murder on a train, a neurotic Jewish financier, a doomed third-rate dancer, a clumsy lesbian journalist - and failed. Actually about gender and lasting damage:
    "why do you do all this for me? I'm not pretty. I guess I'm not clever."

    She waited with longing for a denial. "You are lovely, brilliant, witty", the incredible words which would relieve her of any need to repay him or refuse his gifts; loveliness and wit were priced higher than any gift he offered, while if a girl were loved, even old women of hard experience would admit her right to take and never give. But he denied nothing. His answer was almost insulting in its simplicity.

    "I can talk easily to you. I feel I know you." She knew what that meant.
    "Yes," she said, with the dry trivial grief of disappointment, "I seem to know you too"...
    Heartbreaking in his usual profound manner.
    4/5.



MAR

  • Reread: Gateway (1975) by Frederik Pohl. Hits hard, leaves marks. The same ignoble, epistemically pinched, economically realist sci-fi written by the Strugatskys or Stross. I love it so much that even the Rogerian psychotherapy at its core doesn't annoy me; that even its 90% focus on one spoiled and abusive bastard is a merit of it. Spoilers everywhere. Physics and sin. No shortage of things left to do.
    5/5.



  • Superintelligence (2014) by Nick Bostrom. Like much great philosophy, Superintelligence acts like a space elevator: by making many, many small, reasonable, careful movements - you suddenly find yourself in outer space, home comforts far, far below. It is as rigorous as any work whose topic doesn't exist can be; its author is one of the clearest thinkers I have ever encountered, (and I've been trying quite hard to encounter those). I didn't find this hard to read, but I have been marinating in tech rationalism for a few years and have absorbed much of it at third-hand so YMMV.
    Many of the points made in this book are probably wrong. It is also likely that there are considerations of critical importance that I fail to take into account, thereby invalidating some or all of my conclusions. I have gone to some length to indicate nuances and degrees of uncertainty throughout the text — encumbering it with an unsightly smudge of “possibly,” “might,” “may,” “could well,” “it seems,” “probably,” “very likely,” “almost certainly.” Each qualifier has been placed where it is carefully and deliberately. Yet these topical applications of epistemic modesty are not enough; they must be supplemented here by a systemic admission of uncertainty and fallibility. This is not false modesty: for while I believe that my book is likely to be seriously wrong and misleading, I think that the alternative views that have been presented in the literature are substantially worse - including the default view, according to which we can for the time being reasonably ignore the prospect of superintelligence.
    Bostrom introduces dozens of neologisms and many arguments. Here is the main scary apriori one though:

    1. Just being intelligent doesn't at all imply being benign; intelligence and goals can be independent. (the orthogonality thesis.)
    2. Any agent which seeks resources and lacks explicit moral programming would default to dangerous behaviour. You are made of things it can use; hate is unnecessary. (Instrumental convergence.)
    3. It is conceivable that AIs might gain capability very rapidly through recursive self-improvement. (Non-negligible possibility of a hard takeoff.)
    4. Since AIs will not be automatically nice, would by default do harmful things, and could obtain a lot of power very quickly*, AI safety is morally significant, deserving public funding, serious research, and international scrutiny.

    Of far broader interest than its title (and that argument) might suggest to you. In particular, it is the best introduction I've seen to the new, shining decision sciences - an undervalued reinterpretation of old, vague ideas which, until recently, you only got to see if you read statistics, and economics, and the crunchier side of psychology. It is also a history of humanity, a thoughtful treatment of psychometrics v genetics, and a rare objective estimate of the worth of large organisations, past and future.

    Superintelligence's main purpose is moral: he wants us to worry and act urgently about hypotheticals; given this rhetorical burden, his tone too is a triumph.
    For a child with an undetonated bomb in its hands, a sensible thing to do would be to put it down gently, quickly back out of the room, and contact the nearest adult. Yet what we have here is not one child but many, each with access to an independent trigger mechanism. The chances that we will all find the sense to put down the dangerous stuff seem almost negligible. Some little idiot is bound to press the ignite button just to see what happens. Nor can we attain safety by running away, for the blast of an intelligence explosion would bring down the firmament. Nor is there a grown-up in sight...

    This is not a prescription of fanaticism. The intelligence explosion might still be many decades off in the future. Moreover, the challenge we face is, in part, to hold on to our humanity: to maintain our groundedness, common sense, and goodhumored decency even in the teeth of this most unnatural and inhuman problem. We need to bring all human resourcefulness to bear on its solution.
    I don't donate to AI safety orgs, despite caring about the best way to improve the world and despite having no argument against it better than "that's not how software has worked so far" and despite the concern of smart experts. This sober and kindly book makes me realise this has more to do with fear of others' sneering insinuations than noble scepticism or even empathy.
    4.5/5.


    * People sometimes choke on this point, but note that the first intelligence to obtain half a billion dollars virtually, anonymously, purely via mastery of maths occurred... just now. Robin Hanson chokes eloquently here and for god's sake let's hope he's right.



  • A Devil's Chaplain (2003) by Richard Dawkins. Essay collection from his heyday. His letter to his 10yo daughter is maybe the clearest statement of sceptical empiricism ever, though it also displays the blithe wonkishness that alienates most people:
    Suppose I told you that your dog was dead. You’d be very upset, and you’d probably say, ‘Are you sure? How do you know? How did it happen?’ Now suppose I answered: ‘I don’t actually know that Pepe is dead. I have no evidence. I just have this funny feeling deep inside me that he is dead.’ You’d be pretty cross with me for scaring you, because you’d know that an inside ‘feeling’ on its own is not a good reason for believing that a whippet is dead. You need evidence. We all have inside feelings from time to time, and sometimes they turn out to be right and sometimes they don’t. Anyway, different people have opposite feelings, so how are we to decide whose feeling is right? The only way to be sure that a dog is dead is to see him dead, or hear that his heart has stopped; or be told by somebody who has seen or heard some real evidence that he is dead.

    People sometimes say that you must believe in your deep feelings inside, otherwise you’d never be confident of things like ‘My wife loves me’. But this is a bad argument. There can be plenty of evidence that somebody loves you. All through the day when you are with somebody who loves you, you see and hear lots of little tidbits of evidence, and they all add up. It isn’t purely inside feeling, like the feeling that priests call revelation. There are outside things to back up the inside feeling: looks in the eye, tender notes in the voice, little favors and kindnesses; this is all real evidence.
    Aaag he used to be so wise and grand. (He remains brave and clear, but you don't necessarily want to look through this windows anymore.)
    3*/5.



  • Reread: Tell Me No Lies (2004) edited by John Pilger. An anthology of great investigative journalism, mostly of ignored or neocolonial massacres. (You don't resent Pilger putting his own Cambodia piece in.) Went into this with one eye on Pilger's ideology, but almost every piece is grounded and humane and appalling and beyond the reach of theory to pervert. (Only the Eduardo Galeano rant addresses too many targets at once and fades into zine-ish aspersion. But even that's more than half true.) Gellhorn on Dachau. Cameron on North Vietnam. Hersh on My Lai. Lockerbie. Iraq. The overall target is the powerful who stand by or enable atrocities; Kissinger leers like a terrible wraith from more than a few of these pieces. I cried at this ten years ago and again now and again whenever.
    5/5.






Springs and summers full of song and revolution.
The Popular Front, demonstrations and confrontations,
time that takes you away from yourself and your poetry,
so that you could see them as if from cosmic space,
a way of looking that changes everything into stars,
our Earth, you and me, Estonia and Eritrea,
blue anemones and the Pacific Ocean.
Even the belief that you will write more poems. Something
that was breathing into you,
as May wind blows into a house
bringing smells of mown grass and dogs' barks, -
this something has dissipated, become invisible
like stars in daylight. For quite a time I haven't
permitted myself to hope it would come back.
I know I am not free, I am nothing without
this breathing, inspiration, wind that comes
through the window. Let God be free,
whether he exist or no. And then, it comes
once again. At dusk in the countryside
when I go to an outhouse, a little
white moth flies out of the door.
That's it, now. And the dusk around me
begins little by little to breathe in words and syllables.

*

In the morning, I was presented to President Mitterrand,
in the evening, I was weeding nettles from under the currant bushes.
A lot happened inbetween, the ride from Tallinn to Tartu and to our country home
through the spring that we had waited for so long,
and that came, as always, unexpectedly,
changing serious greyish Estonia at once
into a primary school child's drawing in pale green,
into a play-landscape where mayflies, mayors and cars
are all somewhat tiny and ridiculous... In the evening
I saw the full moon rising above the alder grove. Two bats
circled over the courtyard. The President's hand
was soft and warm. As were his eyes,
where fatigue was, in a curious way,
mingled with force, and depth with banality.
He had bottomless night eyes
with something mysterious in them
like the paths of moles underground
or the places where bats hibernate and sleep.


- Jaan Kaplinski

30/03/2016

Highlighted passages from Bostrom's Superintelligence




Many of the points made in this book are probably wrong. It is also likely that there are considerations of critical importance that I fail to take into account, thereby invalidating some or all of my conclusions. I have gone to some length to indicate nuances and degrees of uncertainty throughout the text — encumbering it with an unsightly smudge of “possibly,” “might,” “may,” “could well,” “it seems,” “probably,” “very likely,” “almost certainly.” Each qualifier has been placed where it is carefully and deliberately. Yet these topical applications of epistemic modesty are not enough; they must be supplemented here by a systemic admission of uncertainty and fallibility. This is not false modesty: for while I believe that my book is likely to be seriously wrong and misleading, I think that the alternative views that have been presented in the literature are substantially worse - including the default view, according to which we can for the time being reasonably ignore the prospect of superintelligence.

Eighteen months ago:
Go-playing programs have been improving at a rate of about 1 dan per year. If this rate of improvement continues, they might beat the human world champion in about a decade.


We know that blind evolutionary processes can produce human-level general intelligence, since they have already done so at least once. Evolutionary processes with foresight — that is, genetic programs designed and guided by an intelligent human programmer — should be able to achieve a similar outcome with far greater efficiency.


One can speculate that the tardiness and wobbliness of humanity's progress on many of the "eternal problems" of philosophy are due to the unsuitability of the human cortex for philosophical work. On this view, our most celebrated philosophers are like dogs walking on their hind legs - just barely attaining the threshold level of performance required for engaging in the activity at all.


Extremely simple neuron models use about 1,000 floating-point operations per second (FLOPS) to simulate one neuron (for one second of simulated time); an electrophysiologically realistic Hodgkin-Huxley model uses 1,200,000 FLOPS; a more detailed multicompartmental model would add another 3-4 orders of magnitude, while higher-level models that abstract systems of neurons could subtract 2-3 orders of magnitude from the simple models. If we were to simulate 1025 neurons over a billion years of evolution (longer than the existence of nervous systems as we know them) in a year’s runtime, these figures would give us a range of 1031-1044 FLOPS. By contrast, China's Tianhe-2 computer, currently the world’s most powerful supercomputer, provides only 3.39 x 1016 FLOPS. In recent years it has taken approximately 6.7 years for commodity computers to to increase in power by one order of magnitude. Even a century of continued Moore’s law would not be enough to close this gap.


Think of a "discovery" as an act that moves the arrival of information from a later point in time to an earlier time. The discovery's value does not equal the value of the information discovered but rather the value of having the information available earlier than it otherwise would have been. A scientist or a mathematician may show great skill by being the first to find a solution that has eluded many others; yet if the problem would soon have been solved anyway, then the work probably has not much benefited the world (unless having a solution even slightly sooner is immensely valuable or enables further important and urgent work).


The existence of birds demonstrated that heavier-than-air flight was physically possible and prompted efforts to build flying machines. Yet the first functioning airplanes did not flap their wings. The jury is out on whether machine intelligence will be like flight, which humans achieved through an artificial mechanism, or like combustion, which we initially mastered by copying naturally occurring fires.


first reflect for a moment on the vastness of the space of possible minds. In this abstract space, human minds form a tiny cluster. Consider two persons who seem extremely unlike, perhaps Hannah Arendt and Benny Hill. The personality differences between these two individuals may seem almost maximally large. But this is because our intuitions are calibrated on our experience, which samples from the existing human distribution (and to some extent from fictional personalities constructed by the human imagination for the enjoyment of the human imagination). If we zoom out and consider the space of all possible minds, however, we must conceive of these two personalities as virtual clones. Certainly in terms of neural architecture, Ms. Arendt and Mr. Hill are nearly identical. Imagine their brains lying side by side in quiet repose. You would readily recognize them as two of a kind. You might even be unable to tell which brain belonged to whom... human psychology corresponds to a tiny spot in the space of possible minds...


whole brain emulation... represents a limiting case of drawing inspiration from nature: barefaced plagiarism... This path does not require that we figure out how human cognition works or how to program an artificial intelligence. It requires only that we understand how low-level functional characteristics of the basic computational elements of the brain. No fundamental conceptual or theoretical breakthroughs are needed for whole brain emulation to succeed.

...emulations would at least be more likely to have human-like motivations (as opposed to valuing only paperclips or discovering digits of pi). Depending on one’s views on human nature, this might or might not be reassuring.






In academia, the rigid division of researchers, students, journals, grants, and prizes into separate self-contained disciplines — though unconducive to the type of work represented by this book —
might (only in a conciliatory and mellow frame of mind) be viewed as a necessary accommodation to the practicalities of allowing large numbers of diversely motivated individuals and teams to contribute to the growth of human knowledge while working relatively independently, each plowing their own furrow.


This strategy [minimising the costs of intelligence by maintaining only enough to fulfill your goals] is exemplified by the sea squirt larva, which swims about until it finds a suitable rock, to which it then permanently affixes itself. Cemented in place, the larva has less need for complex information processing, whence it proceeds to digest part of its own brain (its cerebral ganglion). One can observe the same phenomenon in some academics when they have been granted tenure.


A version of the benign approach [toward creating a politially unified world] was tried in 1946 by the United States in the form of the Baruch plan. The proposal involved the USA giving up its temporary nuclear monopoly. Uranium and thorium mining and other nuclear technology would be placed under the control of an international agency operating under the auspices of the United Nations. The proposal called for the permanent members of the Security Council to give up their vetoes in matters related to nuclear weapons in order to prevent any great power found to be in breach of the accord from vetoing the imposition of remedies. Stalin, seeing that the Soviet Union and its allies could be easily outvoted in both the Security Council and the General Assembly, rejected the proposal. A frosty atmosphere of mutual suspicion descended on the relations between the former wartime allies, mistrust that soon solidified into the Cold War. As had been widely predicted, a costly and extremely dangerous nuclear arms race followed.


A passage which has been tendentiously misread as hitting us with "Pascal's mugging" (i.e. literally incredible moral blackmail):
It might not be immediately obvious to some readers why the ability to perform 1085 computational operations is a big deal. So it's useful to put it in context... it may take about 1031-1044 operations to simulate all neuronal operations that have occurred in the history of life on Earth.

Alternatively, let us suppose that the computers are used to run human whole brain emulations that live rich and happy lives while interacting with one another in virtual environments. A typical estimate of the computational requirements for running one emulation is 1018 operations per second. To run an emulation for 100 subjective years would then require some 1027 operations. This would be mean that at least 1058 human lives could be created in emulation even with quite conservative assumptions about the efficiency of computronium. In other words, assuming that the observable universe is void of extraterrestrial civilizations, then what hangs in the balance is at least 10,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000 human lives. If we represent all the happiness experienced during one entire such life with a single teardrop of joy, then the happiness of these souls could fill and refill the Earth's oceans every second, and keep doing so for a hundred billion billion millennia. It is really important that we make sure these truly are tears of joy



The sacrifice [by a superintelligence of all human life in the service of a greater moral good] looks even less appealing when we reflect that the superintelligence could realize a nearly-as-great good while sacrificing much less of our own potential well-being. Suppose that we agreed to allow almost the entire accessible universe to be converted into [instances of its goal] – everything except a small preserve, say the Milky Way, which would be set aside to accommodate our own needs. Then there would still be a hundred billion galaxies dedicated to the maximization of [goal]. But we would have one galaxy within which to create wonderful civilizations that could last for billions of years and in which humans and nonhuman animals could survive and thrive, and have the opportunity to develop into beatific posthuman spirits.



A [world unified into a political unit] with no more technological and intellectual capabilities than those possessed by contemporary humanity should be able to plot a course that leads reliably to the eventual realization of humanity’s astronomical capability potential. This could be achieved by investing in relatively safe methods of increasing wisdom and existential risk-savvy, while postponing the development of potentially dangerous new technologies.

Given that non-anthropogenic existential risks (ones not arising from human activities) are small over the relevant timescales — and could be further reduced with various safe interventions — such a singleton could afford to go slow. It could look carefully before each step, delaying development of capabilities such as synthetic biology, human enhancement medicine, molecular nanotechnology, and machine intelligence until it had first perfected seemingly less hazardous capabilities such as its education system, its information technology, and its collective decision-making processes, and until it had used these capabilities to conduct a very thorough review of its options. So this is all within the indirect reach of a technological civilization like that of contemporary humanity. We are separated from this scenario “merely” by the fact that humanity is currently neither a singleton nor in the relevant sense wise.



Will the best in human nature please stand up

Before the prospect of an intelligence explosion, we humans are like small children playing with a bomb. Such is the mismatch between the power of our plaything and the immaturity of our conduct. Superintelligence is a challenge for which we are not ready now and will not be ready for a long time. We have little idea when the detonation will occur, though if we hold the device to our ear we can hear a faint ticking sound.

For a child with an undetonated bomb in its hands, a sensible thing to do would be to put it down gently, quickly back out of the room, and contact the nearest adult. Yet what we have here is not one child but many, each with access to an independent trigger mechanism. The chances that we will all find the sense to put down the dangerous stuff seem almost negligible. Some little idiot is bound to press the ignite button just to see what happens. Nor can we attain safety by running away, for the blast of an intelligence explosion would bring down the firmament. Nor is there a grown-up in sight...

This is not a prescription of fanaticism. The intelligence explosion might still be many decades off in the future. Moreover, the challenge we face is, in part, to hold on to our humanity: to maintain our groundedness, common sense, and goodhumored decency even in the teeth of this most unnatural and inhuman problem. We need to bring all human resourcefulness to bear on its solution.

Yet let us not lose track of what is globally significant. Through the fog of trivialities, we can perceive - if but dimly - the essential task of our age. In this book, we have attempted to discern a little more feature in what is otherwise still a relatively amorphous and negatively defined vision—one that presents as our principal moral priority (at least from an impersonal and secular perspective) the reduction of existential risk and the attainment of a civilizational trajectory that leads to a compassionate and jubilant use of humanity’s cosmic endowment.