Should we care for machines the way we do for children? If so, how? And what are the ethical and political implications of doing so? Can speculative science fiction offer insights into how we create and tend to a rapidly advancing reality of artificial intelligence?
In this transcript of the Human Centered podcast, a project of the Center of Advanced Study in the Behavioral Sciences at Stanford University, cognitive psychologist Alison Gopnik and award-winning science fiction author Ted Chiang explore the profound ways that we humans care for one another, and what these practices might teach us about how we care for thinking machines.
Ted Chiang is one of America’s most celebrated contemporary science fiction authors, known for work such as The Lifecycle of Software Objects, “Tower of Babylon,” “Hell Is the Absence of God,” “The Merchant and the Alchemist’s Gate,” “Exhalation,” and “Story of Your Life,” which served as the basis for the 2016 film Arrival.
Alison Gopnik is a distinguished professor of psychology and affiliate professor of philosophy at University of California, Berkeley. She is past president of the Association of Psychological Science and is the author of acclaimed books, such as The Gardener and the Carpenter: What the New Science of Child Development Tells Us about the Relationship Between Parents and Children; The Philosophical Baby: What Children’s Minds Tell Us About Truth, Love, and the Meaning of Life; and The Scientist in the Crib: What Early Learning Tells Us About the Mind.
Alison Gopnik (AG): I am so delighted to be part of this conversation with Ted Chiang. When I started doing work trying to use children’s learning as a model for AI, I had about five different people who sent me copies of Ted’s amazing novella The Lifecycle of Software Objects, including Ezra Klein when I was on his show. When I read it, I was so blown away by the fact that not only was this a wonderful novella about AI, but it was also the best literary account of being a parent and a caregiver that I had ever read. It’s interesting that there aren’t more conversations about caregiving in the literary tradition. We could think about some of the reasons why. But this seemed to me to be the most profound example of a fiction about caregiving and the one that captured caregiving best. It may just be that it’s part of the genre of speculative fiction—by imagining an alternative context in which you had to take care of sentient beings that depended on you, you could understand and think about it more deeply than we do when we’re in the trenches of actually taking care of our own real children.
I spontaneously did something I had never done before in my entire life, which was to write a fan letter to Ted about how impressed I was and how much I love his stories and the work. Ted’s work has been very interactive with the kinds of ideas that we have at CASBS; it’s a wonderful example of science and humanities speaking to one another in this literary way. And as it turns out, there’s two projects at CASBS happening right at this moment that Ted’s work speaks to really beautifully. One is the “Imagining Adaptive Societies” project, which is taking speculative fiction and the strengths of speculative fiction and connecting it to the idea of how to make a better society or a better world. The other is the project that I’ve been working on about the social science of caregiving. I’m trying to think about caregiving not just in the narrow context of, say, parenting blogs but much more generally, as a really essential part of human nature, involving not just the way we treat children, but the ill, the elderly, and even the natural world and the potential world of artificial intelligences. The case of artificial intelligence has made the significance and importance of caregiving particularly vivid and Chiang’s The Lifecycle of Software Objects is the best example I know that demonstrates why caregiving should be so important if we were going to actually have genuine artificial intelligence.
So let me start out with the caregiving part, about caring for machines and caring for humans. Lifecycle is such a wonderful story, because it raises the issue of how you put yourself at the service of another being, and also allow that being to have autonomy. If we’re ever going to have artificially intelligent systems—that’s going to be a basic problem that we have to solve. We also have to solve this problem every time we have a new generation of humans. You articulate that really beautifully in that story. And I was wondering if you have further thoughts as you’ve watched what’s happened in AI and just thinking about care in general.
Ted Chiang (TC): I always feel I need to preface any conversation about AI with a clarification about what exactly we’re talking about, because the phrase “artificial intelligence” refers to widely disparate things. Sometimes it’s used to refer to hypothetical thinking machines, sometimes it’s used to refer to applied statistics, and there’s this unfortunate tendency to conflate the two. I always want to make sure that we are clear about what we’re talking about. In terms of the machine learning programs or robots that we have now, I basically think of them as being comparable to thermostats. A thermostat can be said to have a goal, but I don’t think it would be fair to say that that it has any preferences; it has no subjective experience. You can imagine a machine learning program that you have to train to maintain the temperature of a house, and in a sense, you are teaching this program, but you are basically interacting with a thermostat. And that is the situation that we are in with the existing technology.
AG: I completely agree. One of the things that I’ve always said is: if we said, what we’re studying is the extraction of statistical patterns from large amounts of data, instead of artificial intelligence, we would be describing what we were doing much more accurately, but we would be much less likely to have a broad public relations reach.
TC: Yes, and, of course, companies benefit from this conflation. They always use the phrase “artificial intelligence” because they want to imply that there’s some great thinking machine at work when the product they’re selling is just applied statistics.
But now suppose we’re talking about this more hypothetical idea of machines that have subjective experience. Let’s imagine that we had a machine that had the same or comparable level of subjective experience to, say, a dog. You can train a dog to do useful work, and you can train it using punishments or rewards. The evidence suggests that you’ll get better results if you use rewards to train it, so that is a purely pragmatic reason to treat your working dog well. But there are also ethical reasons for treating your dog well, because your dog has subjective experience. If we posit that at some point we will build machines that have subjective experience, then we will have the same ethical imperative to treat them well.
In that context, if we extrapolate further and imagine machines whose subjective experience is getting closer to that of human beings, then all the ethical dimensions become more complicated. Because a dog will never become a person, whereas a child eventually will become an adult and will have enormous autonomy and responsibilities and an entirely different level of agency than a dog ever will. In that scenario—one in which you are raising or training this machine that might eventually become an autonomous moral agent—you have pretty much the same obligation that you have to your child, and have to wrestle with the same questions that all parents do.
One of the guiding questions for me when I was writing Lifecycle of Software Objects was “How do you make a person?” At some level, it seems like a simple thing, but the more you think about it, you realize that it is the hardest job in the world. It is maybe the job that requires the most wrestling with difficult ethical questions, but the fact that so many people raise children makes it very easy to devalue it. We tend to congratulate people who have written a novel or something like that, because relatively few people write novels. A lot of people have children! A lot of people raise children to adulthood! And what they have accomplished is something incredible.
AG: Just in terms of the cognitive difficulty level, it’s an amazing accomplishment. One of the things that we’ve been thinking about in the context of the Social Science Group is that the very structure of what it means to raise a person is so different from the structure of almost everything else that we do. So usually what we do is we have some set of goals, we produce a bunch of actions, insofar as our actions lead to our goals, we think that we’ve been successful. Insofar as they don’t, we don’t. But of course, if you’re trying to create a person, the point is that you’re not trying to achieve your goals, you’re trying to give them autonomy and resources that will let them achieve their own goals, and even let them formulate their own goals. One way you can think about it is, if you think about the classic structure of economics or utility theory, is that you’re an agent, and you’re trying to accomplish your goals, and here’s another agent who’s trying to accomplish his goals, and you have a social contract where you exchange your utilities. That’s totally different from what happens in a caregiving situation. A caregiving situation is one where you have one agent who has power and authority and resources. And instead of pursuing their own goals, they pursue the goals of this other agent. And even more than that, they let that agent formulate their own goals, figure out what it is that they want to do themselves as the digients end up having to do in The Lifecycle story. I don’t think we have a very good sense in politics or economics or psychology of how it’s possible to do that, or how we actually do that.
TC: There are parents who have very specific goals in mind for their children, and they want their children to turn out a certain way. They will do everything they can to ensure that their children turn out that way, and they believe that they are doing what is best for their children. But they are, in a lot of ways, robbing their children of autonomy. And it’s a very difficult thing for parents to let go of that, because they firmly believe that they are doing what is best for their children. I also have to acknowledge that I’m talking about a contemporary view of parenting. Nowadays, we have this view that being a good parent is letting your child become what they want to become, and helping them become what they want to become, rather than bending them to what you want them to become. And I feel like this is analogous to what Kant said about treating people as either means or as ends. It’s not exactly the same, but there’s something similar going on, where you can think of your child as a means toward your goals, versus thinking of them as an end, where they become their own subject and they’re not there to help you. That’s a struggle, that’s something that people have to wrestle with as part of existing in society. But the tension is much more emotionally loaded when it comes to the parent-child relationship.
AG: In the classic Kantian or utilitarian views, the picture is: all right, you’ve got these two autonomous agents, two beings, both of whom can be out in the world doing things. The question is how they negotiate their relationships. That’s the picture. But the thing about caregiving, and again, the digients are such a lovely example of that, is this incredible asymmetry between the abilities and resources that the carer has versus the cared for.
It’s an interesting paradox. There’s one asymmetry, which is that the carer has much more power than the creature that’s cared for. In The Lifecycle, the humans could just end the program at any moment, there’s nothing to stop them from doing this. They’re the ones who have the power. Yet, in caregiving relationships, that powerlessness of the cared for is exactly the thing that motivates the carers to make these remarkable investments, to make these remarkable altruistic sacrifices—again, as the human parents do in the story—precisely because the other agent is powerless, precisely because the other agent needs them, needs their resources. That’s a really interesting, and very human, set of relationships that we haven’t thought about as much. And if we had something that was real artificial intelligence, not a statistical pattern of extraction from large datasets, even a simple system, you’d have to say, “Well, look, if it was actually going to be intelligent, one of the signs of that would be to be able to formulate new goals, formulate new intentions that aren’t the ones that were just programmed into it.” It seems to me as soon as you have a system that’s like that, the issue is about how do you balance maintenance? How do you balance autonomy? Care is going to be relevant, it’s going to come out.
Language models are not using language in the sense that a linguist means when they say “language.” They’re just processing text tokens, which is an entirely different thing.
TC: You were talking about the asymmetry of the parental relationship with children. That is something that I find philosophically really interesting, because in most of our other relationships with, say, our spouse, or our friends, or even our siblings, and certainly coworkers or people who we have economic interactions with, there is a much higher degree of symmetry. In most of these other relationships, there is this assumption that you are free to leave, that we are both participating in this relationship of our own volition, and we can end it if we so choose. None of those things are true of the parent-child relationship. And there’s also this interesting question of who holds the power in the parent-child relationship? If you ask either one of them, they will probably say the other party holds all the power. The child is incapable of leaving the relationship. If a parent voluntarily leaves the relationship, we pretty much think that’s the worst thing they can do. They are stuck with each other. You would never see a relationship of this type between, say, two autonomous adults. If two autonomous adults voluntarily entered a relationship like this, you would think they were insane.
AG: it’s interesting that part of what happens with committed relationships is that there is this assumption, which is, if it’s really a committed relationship, part of the sign of that is that if the asymmetry developed, you would be committed to taking care of that person.
My husband had open heart surgery this year. And one of the things that was really striking to me is that this extremely dynamic, independent person now is lying in a hospital bed, completely helpless. And the effect that it had on me was, okay, I’m really committed now. This is when love and commitment and loyalty really show up in their fullest form—when you have this asymmetry between the two partners. It’s invisible in the politics and economics and psychology and philosophy literature, even though those very strange relationships are so important and significant and play such a big role in our moral lives.
TC: They are only strange because we’ve normalized this economic model of interaction.
AG: Yeah, that’s another reason. That’s another motivation for thinking about care. It’s odd, because on the one hand if you ask someone, What’s the most important thing in your life? What’s the hardest moral decision that you have to make? What’s the place where your deepest emotions were engaged? They’ll tell you something about close relationships of care. And yet, because these statements are associated with emotion and feeling and women, they haven’t had the theoretical impact that you might imagine.
One of the things that we’ve talked about in the center is this Asian tradition with philosophers like Meng-tzu [Mencius]. In that tradition the idea is supposed to be that politics should really start with those close personal relationships. The task for ethics and politics is how to scale up those close personal relationships of care to the level of a state or the level of a country. You can make an argument that one of the things that the Enlightenment did, for example, was to take the contractual relationships, the “I’m an autonomous person, you aren’t, we’re reciprocally negotiating,” and find a kind of software in markets and democracies for scaling those contracts up to the scale of a country or even the scale of the planet. That’s been a very successful enterprise, but it left the care relationships in the realm of the private and personal and not part of the broader ethical and political world.
TC: Yeah, it’s not the responsibility of the state to engage in care.
AG: It’s strange that this thing that’s so important to us on an individual basis ends up being literally invisible in the GDP. It doesn’t show up in any of the measures of labor or markets. If someone’s doing it, if someone’s taking care in this personal private way, it isn’t in the economy. It’s this strange moral dark matter in our politics, and it doesn’t show up. I can say, as someone who got her first degree and training in philosophy, it definitely doesn’t show up in the dominant Western philosophical traditions.
TC: One of the ways conventional economics ignores care is that for every employee that you hire, there is an incredible amount of labor that went into that employee, just by virtue of them being a person. How do you make a person? Well, for one thing, you need 100,000 hours of effort to make a person. Every employee that any company hires is the product of 100,000 hours of effort that the company didn’t have to pay for.
AG: Yeah, that’s an interesting externality.
TC: They are reaping the benefits of an incredible amount of uncompensated labor. If you had to actually pay for the raising of everyone that you would eventually employ, what would that look like?
There’s a science fiction writer named Greg Egan, and there’s a line from one of his novels that I always like to quote on the topic of artificial intelligence. He says: “If you want something that does what you tell it, use ordinary software. If you want consciousness, people are cheaper.” And that’s very true. People are cheaper, because all the costs of creating people are externalized, they’re borne by someone else. One way to think about the project of artificial intelligence is: Can you create a person cheaply? Can you create something that is the functional equivalent of a person, but that doesn’t require decades of labor? If you can do that, then you have saved yourself so much. The promise of artificial intelligence is a labor force, which is perhaps infinitely reproducible, and which you owe nothing to. And this ties in with the fact that because human beings are the product of decades of life experience, and social relationships, one of the things that comes from that is that we recognize that people are owed things and they deserve to be treated well. This is why I’m super skeptical of current approaches to artificial intelligence, which seem to imagine that if you just cobbled together enough thermostats, then you get a person. You might be able to get some fairly useful tools with that. But I don’t think it’ll produce anything that does what people do.
AG: There is a big project that I’m involved in with my colleagues at BAIR, the Berkeley AI research center. The idea is, what kinds of things could we learn from just empirically looking at how children learn as much as they do, and can this knowledge be implemented in the design of artificial systems. This is not trying to create a person, it’s just asking are there things that we can learn even for relatively simple problems, like getting a robot that could sort nails into different containers.
The two things that come up again and again are that the solution doesn’t lie in how much data you have, or how much compute you have. When you look at children, they’re engaged with the external world. They’re doing a lot of exploration, they’re doing a lot of experimentation, and they’re doing it in a remarkably effective way. Without having a lot of self-conscious knowledge about it. We call it “getting into everything” when a two-year-old is out in the world and exploring, but empirically, when we look at it as developmental psychologists, what we see is that they’re actually performing just the right actions that they need to perform to get the data that they need to make the next discovery to figure out the next thing about how the world works. And when you look at robotics, even just being able to get a robot that can do the very simplest things is far beyond what we can do at the moment.
The other thing is that children are learning by being in social relationships. That is a very simple, obvious thing that kids do when they’re learning, but it’s really hard to build into even a very, very simple artificial system. So the secret is a combination of actually being out in the real world, getting data from the real world, and revising what you do in light of what happens out in the world … The same thing is true about your interactions with other people. And those interactions with other people couldn’t happen unless you were in a social setting in which people cared for one another.
TC: Yes, I completely agree with you. I have long been a subscriber to the idea that any intelligence needs to be embodied and situated. The very first problems a baby has to solve are, How do I move my body? How do I move around in the world? Those tasks are entirely missing from existing large language models. And, as many people noted, the phrase “large language model” is a misnomer, because these models are not trained on language, they’re trained on texts. They’re just looking at a stream of tokens and predicting what token should come next. Language refers to things, to physical objects, and to other people. Because these language models don’t have access to the real world, they have no interaction of any meaningful sort during their training. They are not using language in the sense that a linguist means when they say “language.” They’re just processing text tokens, which is an entirely different thing.
AG: There’s a version you could think of, which is that the large language models are the postmodernist Derridean picture of intelligence come true. When you see that, you realize, oh no, that was wrong all along, you can’t be intelligent if you’re just within the text. Even if you think about it from an evolutionary perspective, there’s a pretty good argument that you start to see brains in the Cambrian explosion and what happens in that explosion is that you start getting eyes and limbs, you start to be able to move and you start to be able to see, and as the great psychologist James Gibson thought, you see in order to move, and you move in order to see. Those two things are interacting all the time. When you start getting creatures that have perception and have action, that’s when you start getting a brain, that’s when you start getting something that looks like real intelligence. That’s all predicated on the fact that you’re in a real world, you’re in a real world that’s external to you, you’re not in this postmodernists text world, and you’re always finding out new things about the world. You’re always getting surprised by the things that you find out about in the world. You’re always changing what you think and changing your representations based on those things that you find out about the world.
You can see there’s a continuity between the simplest organism that has eyes and claws and what we do in science: where we use our intelligence, our ability to do experiments to figure out brand-new things about the world that we couldn’t have figured out beforehand. One of the ways that people in computer vision describe it is as “the inverse problem.” I really like that phrase. The inverse problem is that there’s a world that’s outside of you. It’s giving you data, you’re finding out things about it. But you’re never really going to completely know everything about what that world outside of you is like. The great problem of intelligence, the great problem that brains are designed to solve is: here’s a bunch of photons hitting the back of my retina and bits of disturbance at my eardrums and somehow from that I’m reconstructing that there’s a chair, there’s a person, there’s a microphone, there’s quarks and leptons, and distant black holes. That’s exactly what human intelligence enables us to do. That’s very different from just taking a bunch of data, even a very large amount of data, and pulling out the statistical patterns in that data.
AG: I want to ask you about something that comes up in both Lifecycle and in some of the other stories, which is thinking about the way that human beings are living in time. It comes up in the caring context, it comes up in some of the things that you’ve described, where you don’t know what the outcome is going to be, you don’t know what the possibilities are. Humans live in a world where time isn’t cyclical and isn’t completely symmetrical, but instead there is a single vector of time going forward into the future, and that really influences the way that we function. Part of what you’ve done in your work, especially in “Story of Your Life,” is think about what it would be like if our relationship to time was different.
TC: In our culture, we take for granted the idea that the future lies ahead of us and the past lies behind us, and that we are moving forward toward the future. But those are cultural conventions rather than human universals, and other cultures perceive time differently. Some cultures think the past lies in front of you while the future lies behind you. At one level this makes sense because we know the past while the future is hidden, but at another level it seems weird to us because it might be taken to imply that we are moving backwards toward the future. But maybe we shouldn’t see ourselves as moving at all. Maybe we should see ourselves as standing still and time is flowing past us. Or maybe it’s something else altogether.
AG: It’s such an interesting idea to try and think about what it would be like if you were interacting with aliens who really had a different relationship to time than you did.
TC: The inspiration for that story did not really come out of linguistic or cultural relativity. For that story, I was interested in the idea of the inevitability of loss. I was watching this one-man show where the performer was talking about his wife dying of cancer. There was a point when they knew how it was going to end, and I was very moved and affected by that. And the more I thought about it, the more it seemed like that is one of the things that is part of being human, the fact that we can conceptualize the future. We know what’s going to happen to us in a way that, for example, dogs cannot. And we know that we will suffer losses in the future. One of the things that marks the passage from being a child to being an adult is recognizing the fact that everything eventually comes to an end. How do you cope with that? How do you move forward with that knowledge? For “Story of Your Life,” what I was interested in there was telling a story about someone who has this very specific knowledge of a loss that lies in her future, and exploring how she can she live with that.
This article was commissioned by Caitlin Zaloom.