363 | Chandra Sripada on How LLMs and Humans are Cognitive Cousins

0:00:00.9 Sean Carroll: Hello, everyone, and welcome to the Mindscape Podcast. I'm your host, Sean Carroll. I presume that everyone has heard of the Turing Test, or as Alan Turing himself called it, the Imitation Game, supposed to be a way to figure out whether computers can think. And of course, the difficulty in that is not only what computers can do, but what do you mean by thinking? So Turing took a very sciencey, physical-sciencey, mathematical approach to the problem. He says, I don't know what it means to think, but we can look at what things do. And if you have a computer that does things that are impossible to tell the difference between that and thinking, that is to say the input-output responses of the computer are indistinguishable from those of a person, we should call that thinking. Now, these days we have these LLMs, these large language models, as an approach to AI, and more or less it's clear that they do pass the Turing Test. I know there's some people who argue about that, but I think it's kind of nitpicky myself. I think that there's no question in my mind that they're passing the test as Turing himself would have imagined it.

0:01:12.4 SC: So does that mean that the LLMs are really thinking in the same way that human beings are thinking? And I think there's been an argument back and forth. Some people say, yes, that is what it means. Others say, well, no, actually turns out we have to think harder about what it means to be thinking. And then the other side says, oh, no, now you're moving the goalposts. I thought we agreed on the Turing Test. I'm actually on the side of the goalpost movers. I think it's perfectly okay to say, well, that wasn't a careful enough definition of what it means to think, at least not in the same way as human beings do. And even granting that given a certain set of inputs, the LLMs will produce outputs that are more or less indistinguishable from some kind of human output, there's still a question: Are they doing it because the LLM architecture, in the process of being set up and then trained and fine-tuned and so forth, has essentially rediscovered the mechanisms by which human beings think? Or is it because, and it's an absolutely plausible scenario, they've discovered a different way to have the same input-output mechanisms as human beings, a kind of alien intelligence?

0:02:27.6 SC: And there is some evidence, and I have absolutely been on the side of being impressed by the evidence that says, look, there's questions you can ask an LLM that don't look like human answers. The famous ones are how many Rs in the word strawberry or something like that. And that, to me, was very good evidence that they're not thinking in the same way that human beings are. And a lot of people push back against my view on that, saying, well, all the great computer scientists and leaders of the AI industry are saying otherwise. And that was never especially convincing to me for the simple reason that those people are experts in computer programming and computer science and AI, but not experts in intelligence and cognitive science.

0:03:10.5 SC: So recently, Johns Hopkins hosted a meeting of the Society for Philosophy and Psychology, and some of us had the idea it'd be fun to do a live podcast interview as part of that meeting. It didn't pan out that way. I was traveling at the same time the meeting was happening, et cetera. But we looked through the people who were visiting for prospective good podcast guests, and we found today's guest, Chandra Sripada. He is a philosopher and a cognitive scientist in the psychiatry department at University of Michigan and also an expert on LLMs. Okay, so he's an expert on thinking and on LLMs and is very well positioned to ask and answer the question: Do LLMs think in similar ways to humans do? And he makes a strong case that they do.

0:04:02.9 SC: Often, in many ways, think using the same kind of thinking methodologies that human beings do. And so, I find him very persuasive. I think that he's made a really good case that at least in an important set of ways, LLMs have rediscovered or been coaxed into refinding out the ways that human beings think, using ideas from cognitive science. How do different tests of what happens during the cognitive process match up between LLMs and human beings? So you can tell for yourself whether or not it's convincing to you. It's not the same as saying that LLMs are conscious or responsible moral agents or anything like that, but this is something we should establish in that direction. Cognition is easier to understand than consciousness. So I think this is one of those podcasts that has shifted my credences in important ways. And one should always be a good Bayesian: If more evidence comes in the other way, then they'll shift back, or if the evidence keeps pushing in this direction, they'll keep moving in that direction. But I think it's fascinating that the option, which was always on the table, and I always admitted certainly, that LLMs sort of have, by some way or another, rediscovered human modes of thinking, has turned out to be something that has evidence for it in an interesting way. So I think that makes for a great conversation. Let's go.

[music]

0:05:55.3 SC: Chandra Sripada, welcome to the Mindscape Podcast.

0:05:58.5 Chandra Sripada: Oh, thanks, Sean. I'm glad to be here.

0:06:01.2 SC: Now, we're gonna be comparing two of my favorite cognitive systems, the human brain and LLMs. Most of us are a little bit familiar with the human brain. Of course, we've all heard of LLMs by now, but could you give us a quick overview of how LLMs work? We use them, but maybe we've forgotten what is going on underneath the hood a little bit.

0:06:26.3 CS: Yeah. Of course, yeah. Let me say a bit about that. And I realize that I didn't get to say this already: I'm a longtime fan.

0:06:30.4 SC: Oh, thank you.

0:06:34.5 CS: And so, it's great to be here. And I don't think I've ever heard you at anything less than 1.5 speed. And so it's good to hear you live. You're even more eloquent at this rate. So LLMs, how do they work? For starters, LLMs are neural networks. So there's gonna be a set of nodes and weights that connect them. And the basic processing strategy is when an input comes in, the activations will be multiplied by the weights and get propagated through the network. And if those weights have been tuned correctly, the answer will be a sensible continuation of the input that came in. So LLMs are distinctive in a few ways. And one of them is the approach that they're gonna take is autoregressive. So, "I like to have my coffee with sugar and..." and they have been trained so that massive set of parameters, hundreds of billions, trillions in frontier models, are tuned up in such a way that over the course of internet-scale data, they are making predictions of the next word. And every single time, those parameters have been changed just a bit so their predictions get better and better.

0:08:08.8 CS: And so initially, "I like my coffee with sugar and..." it might be [0:08:17.9] ____ or butter, and eventually you're going to get to cream. So the autoregressive generation, one word at a time, after that one word is generated, it's added back into the prompt as part of the context. That's one thing that's distinctive about them. The training on internet-scale data, that's another thing that's distinctive about them. And maybe it's best to just foreshadow this and say this a bit later, but it's not a big undifferentiated neural net. In the old heyday of PDP, when I was a bit younger, those nets, there is a set of input nodes and a set of weights and eventually, you get to the output. Well, these are transformers and they have a lot more structure. And so with the transformer, every word/token, we don't have to get too much into that, enters as part of a column, and the processing happens in a column-specific way but for the presence of these attention heads that move information from the left over to token positions on the right. And so that's another... It's not the only feature of transformers, but they're a distinctive, highly structured kind of neural net. And so billions of parameters trained on next-word prediction, autoregressive generation, the structure of a transformer, that's a lot of what's going on in a large language model.

0:09:53.9 SC: You know, if I am going to write a book, I do not start at the first sentence and just keep typing, right? I imagine an outline and then I develop it chapter by chapter. I could ask an LLM to write a book. Would it do that? Would it sort of give a big overview and then fill in, or would it just start at the first sentence?

0:10:15.2 CS: Right. So already, and I'm going to try to assemble some arguments that these LLMs process in ways that are reminiscent of or analogous to core processing principles of the human mind/brain. But you caught me already. There seems to be something very different about the autoregressive next-word prediction training and then the generation process where one word is generated at a time and added to the context. What to say about that? I mean, one thing to say already is generating the next word is not incompatible with the model itself during the forward pass, where it has a substantial amount of mnemonic resources and processing capacity to start anticipating what happens later on. So Anthropic has this example where you give it a stem of a rhyme, something to the effect of, "I saw a carrot, I had to grab it."

0:10:56.5 SC: Okay.

0:11:37.8 CS: And the model actually generates the last word of the next sentence, "rabbit," first. And we know this via mechanistic interpretability, where we're peering into the model and looking at its processing stages before it generates the prior words that will culminate in "rabbit." And that's highlighting that next-word generation as a generation method, as well as a closely aligned training objective, is quite compatible with the model generating things that come much later early on. And so that kind of planning can still happen. I'll add another dimension to it as well. We'll come to this a bit later, I think, but these models can use the context window as a kind of scratchpad and generate internal tokens as part of what they call thinking. So if you give it a complex kind of task, the models now will break it into subtasks as part of the initial thing that it'll do. And subtasks can be further broken down into subtasks and delegated to a series of sub-agents and things like that. So it is compatible, remarkably enough, with next-word generation and next-word prediction as the training objective for substantial, staged attacking of a problem, breaking it down into sub-components, looking ahead at terminal parts of the sub-components before next-word generation begins. So next-word prediction is... And the training objective, next-word prediction and autoregressive next-word generation, seems counterintuitive at first, but can actually encapsulate a lot of the ordinary strategies that people use.

0:13:37.8 SC: And is this idea of a scratchpad and multiple sort of swipes at the answer, is that something that human beings have coaxed the LLM into doing, or is that something that the LLMs have sort of figured out for themselves in some sense?

0:13:57.2 CS: Right. And the answer is mostly the LLMs have figured out for themselves, and there is some coaxing involved as well. And so why do we believe something like that? The use of, you talked about the use of the scratchpad specifically, but is this a good time to go into the dual-process distinction?

0:14:25.9 SC: No, we're gonna keep that later.

0:14:27.6 CS: We're gonna keep that for later. Then I'll say this a bit in a very general way. Next-word prediction on internet-scale data gets the model to assemble an astonishing representational landscape and latent strategies for how complex problems like, let's say you tell me that you want to go to Tasmania, the model has latent within it the kind of subtasks that would be needed in order to do that. So all of this just seems to be something as part of the representational repertoire that next-word prediction generates, especially when it's done at scale. Now, when you've got these LEGO pieces, it doesn't mean that you've got that pretty amusement park already that you can build with the LEGOs. There is some coaxing, some supervised training, some instruction tuning, and especially reinforcement learning, where you train the model with evaluative feedback that can emerge from various sources. That tends to take this astonishing representational repertoire that's there, that's latent, and assemble it into useful pieces and organized, goal-directed control.

0:15:48.6 SC: So that's great because yeah, it leads right into what I really want to ask, which is at a very high level, so we'll get into the details later, but to me, the basic question is, look, we've built something that is really, really good at taking in inputs and giving outputs that are remarkably human, right? That could fool anybody if you had the old-fashioned Turing test. And I can imagine two different possibilities. One is that's because the LLMs have basically rediscovered the same mechanisms that are going on in the human brain. Or alternatively, the LLMs have figured out a wholly new way of sounding human, what we might think of as an alien kind of intelligence. And you know that we're relentless anthropomorphizers, so it's gonna be easy to guess that they must be human if they're acting human in this way. So what are the options here before we start advocating for either one? What are the possibilities to help conceptualize what's going on inside the box?

0:16:58.0 CS: Yeah, I think you've laid it out really, really well. We see the outward, we see the inputs, the prompts, and we see the next words come out and they're eerily fluent. And one option is they are fancy autocompletes, they're statistical approximators, they are bullshitters of sorts. They are entities that have memorized a bunch of tricks and heuristics about how to generate next words that sound like people, but under the hood, very little is going on that closely resembles what happens in the human mind-brain. And this is definitely a continuum. There's gonna be various middle positions. And at the other end, a picture that I'll put my cards on the table that I'm gonna argue for a bit more, and I think I'm more inclined to and attracted to, is it just turns out that prediction is incredibly powerful. It is the training signal that is the mother of all training signals.

0:18:15.5 CS: And whereas when I was going in grad school, this was much less appreciated, it wasn't part of the zeitgeist, it has been over the years in cognitive science. It's been taking over center stage as a lot of what happens in the mind-brain is prediction. And a lot of cognitive principles that we thought were innately specified or due to some sort of contingent evolutionary trajectory, they actually are emergent. In cognitive science, dual process distinction or various other cognitive principles that we describe the human mind-brain, they are downstream of prediction. And so, it turns out that at the level of basic core cognitive principles, the LLMs and humans, they identify similar representations, similar procedural techniques, different modes of inferential organization and so forth. And so, there's a lot of similarity there, and it arises downstream of prediction.

0:19:18.5 SC: I like how you call it the mind-brain. That sort of avoids worrying about calling it the brain and people getting upset or calling it the mind and people getting upset.

0:19:28.4 CS: Exactly, exactly. I've been coached by a PR firm not to get into that.

0:19:35.4 SC: So, okay, good. So, the options are out there. Maybe LLMs have come up with a different way of acting human, or maybe there's some form of convergent evolution where the best way of acting human is also what the LLMs found. There is this quasi-anecdotal evidence that has made a big impression on me, I will confess, that the kinds of mistakes that LLMs make seem to be the kinds that no human would ever make. Of course everyone makes mistakes, but there's different kinds of mistakes. Like, famously, the LLMs can't count the number of Rs in the word strawberry until you really teach them. So, how much of an impact does that kind of evidence make on you?

0:20:27.2 CS: Yeah, I think... I'm a total evidence guy. So, yeah, that's a data point. You look at that's puzzling, especially in the next moment it's expounding on parts of general relativity that are very subtle. So, what is going on there? And one thing that we shouldn't do is just stick with these kind of behavioral observations. You're gonna notice various places where the LLMs do not act in very human-like ways. And we shouldn't settle for just looking at outward behavior and counting and tabulating, "Well, this looks a little similar, this looks a little dissimilar." Very quickly, if we stay at that level, it's always gonna remain any one of these possibilities could be live, alien intelligence versus the name I like to give is cognitive cousin for the other version where they're actually much like us. It turns out with the strawberry, these things, they do have obviously a different kind of learning history and their contact with the "world" is exclusively textual via these tokens, which essentially serve as kind of sensory primitives for them about what comes in.

0:21:56.9 CS: So, they don't even have access to the letter level, it's not they don't have access, but they don't typically operate with individual letters when words come in. So there's a very natural explanation for why they can't count the number of Rs in strawberry. Whenever we see these kinds of anomalies, and there are many, an immediate next question we can ask is at the behavioral level, before we even start looking too closely at underlying mechanisms, are there a bunch of ways in which LLMs reproduce behaviorally phenomena that are familiar, that we've documented over the decades in cognitive science? Do they reproduce some of those phenomena? Turns out there are many such cases and many of them are notable. They catch your eye. Can I go through some examples?

0:22:54.0 SC: Sure.

0:22:54.7 CS: Okay. In the area of language, there are a bunch of effects that psycholinguists talk about. So, an example is the center-embedded sentence. So, "A man ran" is easy to understand. "A man that a woman loves ran" is easy to understand. "A man that a woman that a child knows loves ran" starts to become hard to process.

0:23:12.6 SC: Right.

0:23:30.4 CS: That's center embedding. And psycholinguists have identified, for now, I actually won't talk about mechanisms. Let me save that for a little bit later. There's garden path sentences. "The horse raced past the barn fell."

0:23:38.4 SC: Right.

0:23:46.5 CS: The "fell" seems to come out of nowhere. And it's because I'm not saying the horse raced past the barn, I'm saying the horse that was raced past the barn fell. And so, there's an order effect when you do incremental parsing, these garden paths arise. There are other kinds of effects, similarity-based interference, there are depth changes. So, there's a bunch of these kind of effects that psycholinguists have identified. LLMs exhibit those. Another nice and striking one is in serial list memory, a kind of branch of episodic memory. If you give people a list of words and ask them to say the words back, what you'll find is that there's a recency effect. The very last word has an advantage. The very first word has an advantage. There's a lost-in-the-middle effect. Those things in the middle, they don't have that advantage. There's a contiguity effect where if you say a word, the words around it come to mind more easily. There's a temporal asymmetry effect so that if you say a word, the one after it comes to mind more easily. LLMs exhibit all these effects. When you put a bunch of words in their prompt, the first word, the most recent word, are more prominent to enter into processing. There's a lost-in-the-middle, there's a contiguity effect, and then there's the temporal asymmetry effect. I'll give one more example.

0:24:49.7 SC: Yeah.

0:25:15.6 CS: And these... So, visual search has been extensively studied and there's a classic effect, disjunctive versus conjunctive search. So, if I say, "Find the green L among a bunch of red Ls." So, picture that. There's one green L and there's a bunch of red Ls. The green L's gonna pop out because the greenness is a singleton concept that distinguishes that one thing. If I say, "Find a green L among red Ls and green Ts."

0:25:35.6 SC: Hmm.

0:25:55.7 CS: Now, you gotta problem. There's no one feature that distinguishes the thing that you're looking for. And so, people slow down, they do this in serial order, and their time in completing the task is proportional to the number of elements in the array. So, this is classic. What I love about these is cognitive scientists have already written down these effects decades ago. And you can check, do LLMs, in this case, vision-language models, do they exhibit the effect? And they do. All of these examples, we know the mechanisms behind it. We know that garden paths arise due to incremental parses that happen one word at a time that lead to bad parses leading you astray, and then you've gotta go back and regroup. We know the episodic memory effects. We know some of the mechanisms that play a role. And in the visual search, it has to do with what are the conceptual primitives, what is encoded via a compositional code, and what requires feature binding. The fact that you're seeing these non-obvious patterns of similarities in LLMs and people, especially where we know some of the mechanisms that happened in these effects in cognitive science, they point to similar mechanisms being operative in the LLMs and people.

0:27:24.1 SC: So, if I can try to summarize that, the strawberry thing could conceivably be evidence that the LLMs have found a very different way of sounding human, and here, we're showing that the emperor has no clothes with the strawberries. Or mostly, the LLMs are doing human-type cognition and we've just found one little Achilles' heel.

0:27:49.5 CS: I would actually put it in an even more benign way. We knew at the get-go that their sensory evidence, what is analogous to that, is tokenized representations of English language words. They very rarely descend to the level of individual letters. So, we knew that at the get-go. It wasn't supposed to even be a major dimension of the analogy between them and us, that individual letters would be particularly salient in their processing strategies. So, the two lessons I would say are that counting and tabulating at the level of behavioral outputs is probably not gonna get us very far. We need to look mechanistically, and we need to think about which are the mechanisms that we actually care about that are core processing principles for the human mind-brain. Dealing with individual letters in, that's never something that I would elevate to the level of a core processing principle. Whereas some of the others that deal with episodic memory and visual search, compositional codes, and incremental parses, that's bread and butter of what cognitive scientists thought about for a long time. And so the fact that those are preserved are probably more important.

0:29:10.0 SC: I don't want to dwell on all of the different idiosyncrasies of the LLMs, but I guess your answer to the strawberry question doesn't seem to provide an answer to the car wash question. You know this one, right? Do you know this example?

0:29:23.0 CS: I do not know this one.

0:29:24.1 SC: Oh, this is a good example where you ask the LLM, "I am 500 feet away from the car wash. I'd like to get my car washed at the car wash. Should I drive my car there or just walk?" And the LLM will inevitably say, "You should just walk. It's only 500 feet away." Not... With the implication being, but it does me no good to walk because I want to get my car washed there. I need to drive my car there, even though it's very close.

0:29:51.1 CS: You know what's hilarious about that? I was about to say, "You should walk. It's only 500 feet away."

0:29:59.9 SC: You've been hanging out with LLMs for too long, Chandra.

0:30:02.5 CS: Yeah, exactly. Either they've rotted my brain or, unwittingly, what we're seeing is the foibles and idiosyncrasies of them are a little bit like us, at least in some cases.

0:30:17.4 SC: Yeah, I think that's it. Okay. Anyway, I just wanted to get that on the table there. But okay, now, you were very gracious in not going into the dual process theory when you wanted to, but here is a good time to do it because if you're gonna make the argument that LLMs are sort of remarkably human-like in their cognitive strategies, here's one example of how humans work. So explain... Let's not assume that people know what dual process theory is.

0:30:45.3 CS: Sure. Yeah. So what is dual process theory? People often are familiar with Kahneman's 'Thinking, Fast and Slow' and his Nobel... He won the Nobel Prize, and in his Nobel lecture, he revisits his entire amazing research program in terms of dual process theory. And so the idea is you have fast thinking. It's quick, it's automatic, it's intuitive. Results appear to you as a kind of a flash. And then you have slow thinking, which is slow, it's serial, it draws on more cognitive resources. Whereas fast thinking is thought to arise via slow, patient, experience-dependent practice with the environment, a characteristic of slow thinking is it deals very rapidly with sophisticated novel problems. So those are some of the features of fast and slow thinking. An example, if I show you a picture of John Travolta, it flashes out to you, "That's Travolta." It's not as if you reason through it. If I say, "Because so many people attended, they moved the ball upstairs," you know that I'm talking about a gathering of people, not the spherical thing. That's fast thinking. And slow thinking, if I give you, for example, multi-digit mental arithmetic, nothing flashes before your head. There's some sort of procedure. Maybe you as a physicist...

0:32:21.3 SC: No, no, no, no. Arithmetic is not it.

0:32:24.0 CS: Yeah, so that's thinking fast and slow. And it's just a toweringly influential theory. It doesn't originate with [0:32:31.0] ____. So Keith Stanovich and Richard West coined the terminology. And social psychology, stereotyping, they use this theory. Clinical, I'm a psychiatrist as part of my background, substance addiction, ADHD, this is a theory that people draw on. Developmental psychology, adolescents behave the way they do because fast thinking outstrips slow thinking. It's just an explanatory form that is widespread throughout psychology. And so this is an example we better see if we're gonna say that LLMs are like us. Is there something like that in LLMs, and can it help us as cognitive scientists understand this distinction? And the answer is yes. Okay, so in LLMs, there is a really interesting distinction between what people sometimes call in-weight processing, and then there's a pair of, so in-weight processing, so that's W-E-I-G-H-T. I'll come to that in a second. It's very akin to fast processing. And then there's two mechanisms in LLMs that help to undergird that so-called System 2, the slow processing.

0:33:45.4 CS: So I'll start with in-weight. LLMs, via the prediction objective, they acquire massive amounts of information that's compiled into their weights. That's why it's called in-weights. It's a relatively easy pathway from the stimulus information to something that's populated in the LLM. It doesn't require elaborate constructions within their activation space. And I'll say a little bit more about that if that sounds obscure. So, Albert Einstein is German. They just know it. Michael Jordan plays basketball. The state containing Dallas is Texas. If you drop a vase, it will shatter. These are all the kind of common sense, factual things that they know in their weights. There are two mechanisms that undergird their System 2. And the first one, in-context processing, I'm gonna dwell a little bit on it because I haven't mentioned this yet, and I should say this. There's a paper that goes along with some of the arguments that I'm covering today. And that paper, I have a co-author, it's Rick Lewis, professor of psychology and linguistics at Michigan, and my colleague and my friend and collaborator. And so Rick calls, and we'll put the paper up in the show notes eventually, I think, Rick calls this the greatest discovery in the history of cognitive science. So, in-context processing. So maybe that's not hyperbolic enough. What is in-context processing? It's a surprising phenomenon that prediction training installs in the model. Something like a very general routine for extracting patterns from the prompt, generalizing from base cases, learning a novel mapping from instructions alone.

0:35:39.7 CS: When I characterized System 2, I said one of its primary properties is it's the place where rapid, novel grokking of complex, novel patterns suddenly happens. And cognitive science has been at a loss to understand this phenomenon and characterize it in any detail. And in a pair of papers, Radford and colleagues in 2019 and Brown in 2020, the second one is from OpenAI, and Ilya Sutskever is lead author of one of these, and Dario Amodei is the senior author in the other one. These are like giants of the field. Especially in GPT-3, so the 2020 paper, these models, when they're just trained on prediction, they have an astonishing ability to display this in-context reasoning. And an example that was used in a paper that identified one of the mechanisms of in-context processing, I'll give that example. You say, "Marie Curie is..." Well, I mean, Marie Curie, she is the discoverer of radium. She's an amazing scientist. Those are the things that come to your mind. Maybe that's in the weights of the model. But if I say, "Albert Einstein is German, Mahatma Gandhi is Indian, Marie Curie is..." You go, "French." Some may know that she was born in Poland.

0:37:06.5 SC: Polish. I was gonna say Polish, yeah. I know too much about this one.

0:37:11.0 CS: Well, the computer scientists that use this example, they may not have even known that she was Polish, so they just assume the answer is French. The model gets the idea that there is a pattern being requested. And they actually study this in a lot of detail. They use hidden Markov models, mixtures of them, and they show the model is doing something like implicit Bayesian inference. It's got a hypothesis about what task is being asked. It's using the first example, you know, the likelihood. It's multiplying that, presumably, by some sort of prior to arrive at a posterior of what kind of task is there and what kind of completion is the best for this circumstance. Others have likened this to more like gradient descent happening within the model. I want to emphasize, this is happening after the weights are frozen.

0:38:05.4 CS: So what do I mean, gradient descent? Gradient descent is how these models are trained, then they're shipped, and then the weights are frozen. What do you mean, gradient descent? Well, the models have activations. Those activations are changing. Those activations are what are happening on the nodes when you multiply the nodes by the weights. Well, the model has compiled within it machine learning routines like linear regression and nearest neighbor and Bayesianism, and they can understand very complicated patterns in their activation space. That's what this in-context learning is, and it arises via very general prediction training. As a cognitive scientist, where we have struggled to articulate, "What is this System 2?" The dirty secret of cognitive science is we don't actually have theories of things like System 2 or related terms. What we have is theories of, when does it develop during adolescence? What lights up when you put people in the scanner? We don't actually have mechanistic, process-level theories of a lot of how this stuff works. And here we have an artifact that's displaying the phenomenon, and we can mechanistically interrogate it, and we can understand the grab bag of routines that underpin it.

0:39:29.1 CS: So this is startling, this in-context learning, and it's very System 2-ish. There's another System 2 mechanism that they have, and that's where you're using the internal, you're using the context as a scratchpad, and you're doing extended serial reasoning within it. So I'll say a little bit about this quickly, borrowing an example from Melanie Mitchell. So you give a problem like this, "Julia has two sisters. How many sisters does her brother Martin have?" Well, you say, "Oh, two." Well, you just told me she has two, there's two sisters there. Well, Julia is a sister.

0:40:09.4 SC: Yeah.

0:40:09.8 CS: And so if you give this type of problem to a model that is not using chain of thought, it's very likely to say two. But if you allow it to generate internal thinking tokens that are not displayed to the user, that end up in the context, that influence the next steps that the model takes, it can reason through and says, "Okay, so Julia is a girl. She has two sisters. That means there's three girls in the family," and so forth. So you can show that a model that has access to this chain of thought scratchpad, models like the ones that won IMO gold recently in 2024, that this amplifies the inferential power of the model. Okay. And so where we begin is dual process distinction. It's a towering distinction in human psychology. And what we see is these models have in-context processing and they have chain of thought, and between the two of them, they approximate, and they also have the in-weight, which corresponds to the automatic.

0:41:00.7 SC: Yeah.

0:41:13.1 CS: And so you're ending up with a distinction in the model that mirrors that in people. And you ask, well, is there clear evidence that this in-context and chain of thought that actually is much like what happens in the human mind-brain? And yes, there is evidence. And there's reaction time evidence. My own lab has a study using paradigms like the Stroop task that illustrate that there is a very tight correspondence between what is happening in the model and what happens in the human mind-brain, for which we often invoke the System 1, System 2 distinction.

0:41:57.5 SC: So I don't want to gloss over it too quickly. Are you saying that LLMs have discovered Bayes' theorem?

0:42:09.5 CS: Yeah. You know, what to say about that? So it has been widely known that LLMs approximate the posterior. So you've got a context and then there's a next word, and there's a conditional distribution, and they approximate that conditional distribution very well. And a Bayesian would say, "Here's the normative way to do it. You have these factorized representations of hypotheses, you've got a prior, and you multiply it by these likelihoods." It turns out that you can approximate the Bayesian posterior, or the correct conditional distribution, not by explicitly going through a factorized representation, but in a sense by modeling it or learning it, by shortcutting your way through what are the features present in the context. Can I stick those into a powerful function approximator and get something Bayesian posterior-like? Let's call it amortized inference. And it's a statistical principle that's been known in the machine learning community for decades, that you don't have to go through a Bayesian calculation to get to the Bayesian posterior. You can amortize it via the very long learning process. That's what these things are probably doing. And it's not simply that they approximate the Bayesian posterior. That's not actually what in-context learning is. It's they can take a novel situation and map it into a Bayesian approximator in their activations after their weights have been frozen. So the Bayesianism doesn't just happen during that extended predict-the-next-word internet-scale training phase. The Bayesianism penetrates at the level of, after their weights are frozen, how they process.

0:44:01.5 SC: Yeah. And that's very plausibly what is happening in the human brain too. We're not born with Bayes' theorem imprinted on our neurons.

0:44:10.3 CS: Yes. And Sean, I want to say just a little bit more before we leave this topic, because I get so excited about this dual process distinction. To me, this is screaming, "Hey, look at me, I'm an LLM. Look at me, look at me. I'm not so alien," to me.

0:44:30.3 SC: Yeah.

0:44:31.3 CS: I have studied Stroop tasks, they're called conflict tasks in cognitive science, for the last 20 years, since I was a graduate student in philosophy of cognitive science. There's just a laboratory where cognitive scientists spend a lot of time in trying to understand, what are these two processes that are competing and things like that? In LLMs, you can find places where, so the prompt we use is, "The crayon is red, so you say the crayon is..." The model wants to say "red." That's its automatic compiled disposition. Of course, I told you the crayon's red, you're gonna say it's red. Then we append a kind of rule-based prefix that reverses the mapping between red and blue or any pair of colors. What we find... And that taps in-context inferential reasoning, what we find is causal pathways through the model where the automaticity resides, causal pathways through the model where the in-context processing that opposes the automaticity arises. You find classic congruency effects that you find in conflict tasks. You find what's called the congruency facilitation effect, another effect. You find the congruency sequence effect, another effect that cognitive scientists have described. You can fine-tune the model to ramp up automaticity, and you get the predicted effects. You can impair in-context processing using a human-like manipulation where you burden and tax the model by a very long rule, exactly the kind of thing that would mess up a human. And you find that in-context processing is weaker, and it selectively affects the incongruent condition. The whole thing resembles people. And so, I love this example because this is just an entity, and we're doing this on Gemma 2B that hasn't even been instruction-tuned, so it's a small model, it's just predict-the-next-word that gets all of this infrastructure there.

0:45:40.1 SC: Right.

0:46:29.6 CS: If you asked me in 2010, "Where do these dual-process effects come from?" I'd say, "Well, evolution gave us a reptilian brain, and it installed this other infrastructure of higher cortex, and it's all innate." And I was a student at Rutgers where Jerry Fodor was there, and so I would also throw in, "We'll probably never figure out how the System 2 part ever works." I would say all those things. All of that is wrong. Prediction objectives looks like it sets up this entire infrastructure. Wow.

0:47:05.1 SC: Okay. So one way of rephrasing that, again, correct me if I'm wrong, is, look, of course there's this superficial similarity between responses from LLMs, responses from humans, and therefore, on the one hand, you're tempted to anthropomorphize them, on the other hand, you want to resist that temptation. But you're saying that it goes beyond that. There are much more subtle aspects of human reasoning that the typical person on the street doesn't even recognize, but the highly trained cognitive scientist is well aware of, and we're finding those in the LLMs also.

0:47:39.3 CS: Exactly, exactly. And with the dual-process distinction, that's one of those, though, where the man on the street actually does recognize this. 'Thinking, Fast and Slow' is influential in part because it resonates with something that we know from introspection. So this is one of those where we can look inside and see this distinction. Yeah.

0:47:58.4 SC: And at the danger of derailing, how confident are we that it is a truly dual-process theory versus just a multi-process theory? Is it truly just two scales, or are there different processes that happen on all sorts of different scales?

0:48:17.2 CS: Yeah. I mean, the in-weight/in-context distinction, what is it? One thing I will say is that cognitive scientists debate this question, and one of the weaknesses of dual-process theory, any cognitive scientist gets frustrated, is a lot of it operates as a list of adjectives. Well, one of these processes is fast, automatic, effortless; one of these is slow, working-memory dependent. And you've got these adjectives, and hey, where's the, where is the mechanistic details? Without the mechanistic details, it's harder to know how you count.

0:48:52.2 CS: What exactly is the theory committed to and so forth. Now you have an artifact, that is the LLM, and we understand a bit more how in-context processing works versus in-weight. I would describe it much more as a continuum. Everything that an LLM ever does requires its weights and it requires activation. But there's a much more direct link between the stimulus and the response via weights that already have most of the response within it in the case of in-weight. So, "Michael Jordan plays basketball." There isn't an elaborate structure that needs to get built in the activation space in order to get to that response. But if you have more complicated prompts, like that Albert Einstein is German and Mahatma Gandhi is Indian, so Marie Curie is, you have to build up a structure in the activation space, something that infers what is the task, applies that task to Marie Curie, extracts what that response is. The activation space is now crowded with a lot of structure. It's very prone to interference and so forth. So while it is a continuum between in-weight and in-context, there are definitely poles anchoring each end. And so, you're now in a better position to start to understand what this distinction really means because you've got very clear mechanistic hypotheses.

0:50:27.1 SC: So while we're digging into, we're trying to figure out if the LLMs are thinking in very human ways by first thinking about how humans think, and the dual-process theory is one of the examples there. You had another example, which I'm not at all familiar with. I knew about the dual-process theory, but you bring up the idea of production systems, an idea due to Emil Post. And I have no idea what's going on there. So tell us what is going on with that.

0:50:54.4 CS: Right. So now, we've moved away from inferential organization to kind of an even more basic level, kind of computational organization. So at the very beginning, I said, look, transformers, they're not your vanilla, they're not grandpa's neural nets. They have a lot of structure. And the way that they operate, and I'm gonna be getting to it, involves layer-wise transformations that differ from traditional neural nets. And many people in computer science would be very familiar with how the transformer operates, and they know exactly the history that led to that. What they may not know is that in cognitive science, we've been there before, in a sense. And so that's the whole production system. So I'll say a little bit about that. Okay. Production systems are a very important modeling framework in computational cognitive science, and they're used extensively. There's a huge body of results. And so what exactly are they? Well, a good starting place is people are familiar with ordinary, the way ordinary digital computers on your desktop, how they work. They operate with sequential instructions, a fetch-execute cycle. And so you get an instruction, you decode it, you execute what it says, and then you move on to the next instruction in the sequence. And that's a very potent, general framework for computation. Production systems operate with something called a recognize-act cycle. And so the way things are set up is there is a state representation, a kind of persistent memory, and then there is a population of conditionals. And the form of those conditionals is, "If the state has so-and-so features, then do one or more actions." And these, in computational cognitive science, they're all discrete, and I'll loosen that a little bit later.

0:52:56.3 CS: And an analogy that might be helpful is you could program a robot to make tea by telling it exactly, walk into the kitchen, grab the kettle, fill it up, tell it exactly what to do. And here's another way to do it. If the kettle is empty, fill it. Okay. If the water is cold, boil it. If the kettle water is hot, pour it on the tea. So it's reading in the state of the environment and it's executing one or more of these conditionals. And what's interesting is this other way of doing it, this you've got a state and you've got conditionals, it just seems more natural. It's robust. So let's say I walk in there, I grab the kettle and I fill it with ice cubes and the water's cold again. Actually, the robot knows what to do. It's going, "If the kettle water is cold, boil it." I didn't even think when I programmed it that somebody was gonna do the ice cube trick. But cognitive scientists noticed that fetch-execute is just not gonna be a good framework for modeling the human mind-brain. And they gravitated towards the production system, that is the state representation and a population of conditionals.

0:53:37.9 SC: So the distinction is between just a step-by-step algorithm, do this, do this, do this, versus a continual give-and-take with the environment and what should I do next?

0:54:19.6 CS: That's right. And but the environment here is encoded in the form of a state representation, which is the canonical place where our hero, the robot, will be looking in order to know which conditionals to apply.

0:54:27.1 SC: Right.

0:54:34.7 CS: And you know, way back in logic or whatever, we learned these different approaches to universal computation. And Emil Post, way back in 1940, people have heard about Turing, but this state representation population of conditionals is another complete framework for any computable function can be expressed this way.

0:54:47.8 SC: Good.

0:55:00.9 CS: But cognitive scientists very much like this production framework, and it became the dominant modeling framework in cognitive science. And the analogy is really with kind of working memory is where your state is stored. And a lot of what your brain houses in long-term memory is a lot of this population of if-thens. And you have learned over time. And so you can model sentence parsing in this production system framework. In fact, Rick Lewis, my co-author on some of this work, and his former student, they have a very influential production system model of how parsing works. It predicts reaction time, it predicts center-embedding problems, it predicts similarity interference effects. There's theories of working memory retrieval, there's theories of multitasking, all expressed in this production system framework. So the take-home lesson there is it is a good, powerful, probably a leading influential theory in cognitive science that the mind operates with kind of a central state representation and massive populations of conditional rules. It's a good approximation of the computational organization of the human mind-brain. Okay, got that.

0:55:42.3 SC: Yeah.

0:56:23.8 CS: And I'll add some names like Allen Newell and John Anderson. If your viewers want to know where to read, go read these folks. 'Unified Theories of Cognition,' beautiful book. John Anderson's book, 'How Can the Human Mind Occur in the Physical World?' 'Rules of the Mind.' You know, these are, read all about it. This is great stuff. All right, so we got that in place. Meanwhile, far away in another part of town, this is in computer science, a little Bob Dylan reference there for anybody that's interested in that.

0:56:56.7 CS: The computer scientists are struggling trying to create the neural nets that will go deeper. They want more intermediate layers. And he and his colleagues, in a famous paper called ResNet 2016, they find that you're not getting very far just putting more layers. And shockingly, you're actually doing worse on the train data, not just on the test data because of overfitting. You're doing worse on train. And they show there's a way to improve this. Normally each layer, the weights learn a transformation of the previous layer. So layer L plus 1 will be some function f of L, the previous layer. What you need to do is you need to propagate the previous layer up to the next one. So f of L plus 1 is going to be X, the previous layer, and X is just what the or L, the previous layer, plus some function of the previous layer. What that forces the model to do is not learn to rewrite the entire representation. It's going to let that state persist across layers. And what each layer will do is make a small amendment, some small change that's conditional on what features are present, what representations are present at that previous layer. Okay. So the thing that's persisting is called the residual stream. And that's one of the key design components of the transformer. So this ResNet paper, by the way, is actually cited more than Attention Is All You Need, the 2017 paper. Both of those are top 10 papers of all time.

0:57:50.3 SC: Oh my goodness. Yeah.

0:58:43.9 CS: And the other thing that the transformer paper does is it makes explicit that the model is going to have to learn conditionals. And how does it do that? People may be familiar with KQV. That's the linear algebra equation for attention. And what that's saying is that the degree of match between something called a query, so at every token you have this representation, you map that into something called a query. Every other token to the left is going to get mapped into something called a key. And you ascertain the degree of match between key and query. And that number, a scalar, you're going to multiply that by another variable called a value, which is a mapping from each token position to the left. So you're going to scale up and down the amount the value gets added by the degree of match between key and query. Okay, what did I just say there? I said, if key matches query to that extent, add value. I formulated in linear algebra terms a conditional. It's a graded conditional. It's not, "if this feature is present, then add this feature," which is how the cognitive scientists were doing.

1:00:00.2 CS: Why do it as graded conditional? Well, you get a lot more expressivity, but you can learn the thing via gradient descent, and that's the key. And if you can train something at scale with gradient descent, which you can with prediction, now you're off to the races. You've got a production system-like architecture that's learning the population of productions. It's not just attention that operates this way. The MLP units, which are actually where most of the parameters in the model are in the MLP units, they have two matrices. And so the rows of the first matrix contain features. If they match the residual stream to that extent, you add the features in the column of the second matrix. The exact same formulation. Ultimately, what transformers are doing is tons and tons of dot products between two vectors. That's a scalar which is the degree of match between those two vectors, and then you add the third vector.

1:00:58.6 CS: That's 99% of what happens in a transformer, a lot of that. So that's this production system idea. These things are involved in exactly what the cognitive scientists had figured out. These people independently in another part of town had figured out you're gonna need a structure like this, a residual stream and a population of conditions. And so my co-author Rick, he actually looked at transformers using that old ACT-R production system paper, looked at whether, hey, do transformers, do they predict a lot of these same effects that we got from our old paper? Sure enough, they do. And on and on. There's various lines of evidence that these are production system-like. They're not inscrutable, at least at the level of computational organization. We understand the strategy being used in these things.

1:01:51.8 SC: So in multiple ways, tell me whether this is an exaggeration or not, the LLMs, bless their hearts, just trying to predict what's gonna happen next, have reinvented strategies, cognitive strategies that are well used by human beings.

1:02:08.4 CS: Okay. In the dual process that works. Here, there's a small amendment. The LLM is hard-coded, is architected with the residual stream. It didn't learn that.

1:02:13.9 SC: Okay, good, good.

1:02:19.4 CS: In fact, I think what he and colleagues showed in their ResNet paper is, at least with the dataset sizes they were looking at, it can't learn. It's stuck. In principle, it could learn the needed transformations. It could learn that you need to make a little amendment rather than rewrite the whole thing. It just gets stuck in parameter optimization space and it just can't learn it. So you architect a bunch of these things. But the convergence is happening with these computer scientists are making these design choices, God bless 'em, not because they're looking over the shoulder of Alan Newell and John Anderson. They don't know who those folks are. They're just reinventing what Alan Newell and John Anderson figured out. Let's have this kind of state representation, population of conditionals. They're kind of reinventing that strategy.

1:03:05.0 SC: I mean, you've mentioned a lot the importance of prediction as thinking about what the LLM does. Maybe that's the right way to think about what brains do. You know, we've had Karl Friston on the program. There's certainly a school of thought out there that says that the right way to think of the brain is as a prediction machine, the Bayesian brain, the free energy principle, things like that. So, I mean, maybe fill in for us a little bit. Is that an accurate representation of what's going on? Could it have been different than that? Or what are we learning by saying that we are prediction machines?

1:03:46.7 CS: Yeah, that's great. And it's a little bit of jujitsu because one of the things that people say when they say LLMs are not like us is, "What you've done is you've taken something and you're bathing it in prediction. It's predict the next word on internet-scale data. And that's so unlike what people are up to." And that seems to be plausible until you realize that in cognitive science, prediction has been moving to center stage way before LLMs came on the scene. So Friston and Andy Clark, the philosopher, they attribute their research program to Helmholtz in the late 1800s. And so the idea that the mind is continuously predicting has an old history. And in perception, there's these predictive hierarchical coding models that are very influential. And in the hands of Clark and Helmholtz and Friston, they generalize this to just core cognitive principles. What we are doing ubiquitously in perception and cognition is predicting.

1:05:03.5 CS: In language specifically, I've been teaching Intro to Cognitive Science and we always show Kutas and Hillyard 1980. Since 2010, we've been doing this, me and Rick Lewis. And you give people a sentence like, "He spread the warm bread with..." You expect butter, but the word there is "socks." And that old paper in Science shows, a couple hundred milliseconds, people, their EEG is red alert, red alert. And so people are tracking incongruity. And the way they do it is it's a deviation from what's predicted. In 2010, we didn't really think of prediction as a ubiquitous principle in language. Yet, Robert Hale and Roger Levy and these great thinkers were installing it there in language. And then other people were saying it's a very general theory in cognitive science. And then along come LLMs. And what they're doing is they've got all these parameters, they make a prediction, and that allows them, the deviation between observed and predicted allows them to make intelligent revisions to their parameters via chain rule, gradient descent, and things like that. Prediction may be the mother of all training signals. There is just nothing that can give you the density and the high quality of prediction. Yann LeCun has this cake analogy, and there's something to this. You got this massive cake and the main stuff in the cake is prediction. That's how the representational infrastructure is coming from there. And then the icing is a little bit of supervision. And he says the cherry on top is RL. And there's something to this.

1:05:58.1 SC: RL, reinforcement learning.

1:06:52.7 CS: Reinforcement learning. And what RL is gonna do, it's not that, I'd actually put it a little bit differently. It's not that RL is just a little cherry. It's everything from prediction needs to be bolted in place first. RL would never get you there by itself. But once it's there, RL is what's gonna take the LEGO blocks and assemble them into something attractive. And so there's this kind of division of labor, but prediction has a primacy here. And it's a deep convergence between us and LLMs. The centrality of prediction, that's gonna be a deep convergence.

1:07:29.4 SC: Do you know about epsilon machines?

1:07:32.8 CS: I'm not familiar. Tell me.

1:07:35.2 SC: I'm just wondering if it's relevant. It's an SFI kind of concept. Jim Crutchfield and others have developed it, Cosma Shalizi. Basically, they're trying to characterize the complexity of a predictive process of something that, given a string of letters, will predict, or a string of symbols, will predict what comes next with respect to how well you could possibly do. So if the process is just a whole string of zeros, the easiest, the best you can do is predict another zero. That's very easy. But if it's completely random, a coin flip, it's also pretty easy. All you have to do is flip a coin. Whereas if there's some structure there and some complexity, the epsilon machine might require more entropy, is how they characterize it, to really know from the previous string what to predict next. And I'm wondering if that kind of characterization of predictive difficulty would be relevant here.

1:08:36.4 CS: Some people say that large language models are memorizers, and that's not possible because of the combinatorial explosion of the space of possible questions and answers. Can't memorize that. So they are exploiting regularities. And then there is a question of what is the complexity class of the regularities that they are exploiting. And now, we're entering in territory that I'm not that familiar with, and I wouldn't be able to speak with much authority of what complexity class we're dealing with. But it does raise the following issue, which is an embarrassment for my position that they are like us. They require internet-scale data to reach kind of human-level fluency. They do better than us in certain kinds of world knowledge, and they're worse in other areas. But the human child may need, let's say, 100 million tokens. They're gonna need 100 to 300 billion tokens. So we're talking three, four orders of magnitude more. So the LLMs are much less sample efficient.

1:09:25.2 SC: Yeah.

1:09:43.2 CS: There is a question that is probably in the vicinity that your Cosma Shalizi and those people with much bigger brains than I can figure out. There are ways probably that LLMs can do better. And humans are an existence proof. And so let's not get carried away with the fact that they need internet-scale data, but a child needs a much smaller data set. The way that we should think about that is, the structure I would impose on that observation is, first of all, let's distinguish, and I'll use Locke's vivid phrase here, how the mind is furnished. And he asked, he said it was experience. Maybe he's more right than he realized. LLMs don't furnish their mind with the same efficiency that people do.

1:10:07.7 SC: Right.

1:10:32.3 CS: And it's an important observation. But the end state, the mature state, the configuration of tables, chairs, and shelves that they reach may very well be much like us. And so let's first pay heed to that. They may be very informative to cognitive science, even if their mind gets furnished in a different way, because the eventual furniture and their arrangement is very much like us. What would it take to get them to be much more sample efficient? Maybe the old-fashioned nativist. That is, in cognitive science and philosophy, we use nativist not to talk about immigration, but to talk about how rich is the innate structure. Maybe some story like that is true. But what I would bargain on is, and we touched on this earlier, [1:11:21.5] ____ colleagues found you can't train a neural net with multiple intermediate layers easily just on the data. It starts to collapse. You need to give it a head start. Very simple thing, the L+1, the layer+1, is not going to be just some function of the previous layer, but it's going to be the previous layer plus some function of the previous layer. That's all they did. A little tweak like that makes something unlearnable before, which is the limiting case of sample inefficiency, learnable. Could there be some of these tweaks that give us three or four orders of magnitude? That ain't no thing. You and I know. Three or four orders of magnitude? You're a physicist. That's a joke in your neck of the woods.

1:11:22.2 SC: It is.

1:12:12.0 CS: So we don't know whether the nativist story is correct or whether it's a tweak. It's an engineering twitch that if we, a small one, that would give us an order of magnitude here, another tweak, another order of magnitude. And now, we're looking at a very close analogy between us and them, not simply at the level of the furniture arrangement that's eventually reached, but how the mind gets furnished as well.

1:12:36.4 SC: How much of our ideas about what the LLMs are doing is coming from querying the LLMs many, many times versus sort of opening up the box and looking inside? My impression is this is sort of a notoriously difficult problem, but people are making progress on it.

1:12:55.5 CS: Yeah. So opening up the box, that gets directly to the heart of mechanistic interpretability. And so that is an exciting field. I have a 12-year-old, I tell him, you know, who's the Dustin Hoffman in 'The Graduate?' Like some douchey guy comes up to him at a party and says, "plastics." I'm the douchey guy and I tell my son, "Mechanistic interpretability."

1:13:16.6 SC: Okay.

1:13:23.4 CS: I mean, that's the future, right? I mean, you figure out what these models are doing. And so, there is a lot of mechanistic interpretability work. And in cognitive science, we have very rich theories of what kind of features are being tracked, let's say in a linguistic processing setting, grammatical number, relative clause boundaries, it goes on and on. And you can use mechanistic interpretability techniques such as train a classifier to grab the weights at some layer of the model and see whether you can decode the presence or absence of that feature in the prompt. So mechanistic interpretability, we can peer inside and get a handle on what features and representations the model is using, and then we can causally manipulate it to close the circle and make sure, yeah, when we change this feature, the model's behavior changes as predicted. And a lot of that work does reveal phenomenon going on in the model that is reminiscent of what happens in the human brain. I'll point to my colleague Rick Lewis' work looking at transformers. They look at the entropy of the transformer's heads and how it's distributing its attention. And they have from linguistic theory places where the attention will be more distributed because cognitive scientists postulate there's gonna be candidate parses that could receive attention. And they look and they find, well, the entropy is higher there and the human reading times are longer there. So there's these very powerful methods to say on a moment-by-moment dynamic basis, the model is doing things that resembles what people do.

1:15:09.1 SC: So let's just... I know exactly what you mean when you're talking about the entropy here, but it might be there's different notions of the word entropy. Basically, it's a way of, in this case, quantifying uncertainty, right? Like the thinker, whatever it is, is keeping open multiple possibilities until things resolve themselves. That would be high entropy. Whereas if the thinker is pretty sure where you're going, that's low entropy. Is that right?

1:15:35.1 CS: That's exactly right, yeah. And so to make it concrete, take a sentence like, "The problem in the classrooms was solved by..." The word "was" is talking about "the problem." And in fact, there isn't another candidate there. "The classrooms" is in a prepositional phrase and by linguistic theory, it's not eligible to be the subject of "was." If I give you a grammatically incorrect sentence, but people understand grammatically incorrect sentences all the time, "The problem in the classrooms were solved by..." The only eligible thing is "problem." But people struggle there because "were" is plural and they look at "classrooms" as a possible, as a candidate, and the reading times are longer and it's all messed up. In LLMs, the attention, which is looking at the previous words, in the first case, the attention would be sharply positioned over "problem." In the second case, where you got that grammatically incorrect, it's looking at "classrooms" as well and there's confusion.

1:16:14.6 SC: Yeah.

1:16:54.0 CS: So it's a nice correspondence between the two systems. There are other ways of getting at this correspondence as well. There's so-called representational similarity methods, alignment methods. So I do neuroimaging as well. And so you can give people stimuli while they're in the scanner and then you've got their brain activations. And you can give the model the same stimuli, and you can grab a layer from the model, and you can predict the brain activation patterns across the different stimuli. And if you've got a well-configured null hypothesis, preferably by various kinds of non-parametric methods where you scramble the data in various ways, you can see, is the model's weights predicting brain activation patterns more than you expect by chance? And there's now this entire body of results saying in vision, in auditory processing, in language processing, yeah, you're getting these correspondences. It's much more than you expect by chance. People will say all sorts of things. Every literature is mixed, but the weight of evidence says yes. And the larger, more capable models tend to be more correspondent with the human brain. And that's really, really interesting.

1:18:12.9 SC: I mean, this is... You've given us a lot of good evidence for this kind of, as we were discussing at the beginning, evolutionary convergence between modes of thought or modes of cognition, anyway, in LLMs and in human beings. I could invent reasons why that convergence might have happened, but what do the experts say? Do we have theories, hypotheses for explaining why, given the constraints of the problem, this is the solution you would have ended up with?

1:18:46.0 CS: Good. Yeah. So one pair of theorists that... Everything here is going to be a little bit speculative.

1:18:50.3 SC: That's okay.

1:18:54.5 CS: But, given that caveat, one pair of theorists that have lurched forward into the wild are Dan Yamins and Rosa Cao. Dan is a computational neuroscientist and Rosa is a philosopher of neuroscience. And they have extensive collaborations over the years and people should read their stuff. It's awesome. And they have a view they call contravariance. And the observation there is... The theoretical position there is that, look, for easy problems, you're gonna get a lot of different ways of solving them. So you could have a rhesus macaque, a human, and a neural net that are solving some simple visual classification problems, and their internal representations may be quite different. They have different semantic primitives or features and different procedures. As the problem gets harder... Often this is the sheer generality of the problem they need to solve, which is the notion of hard. You're not classifying 10 things; now you're gonna classify a million things. Well, the space of computational solutions becomes much more limited. And there, you expect much more correspondence between the rhesus macaque, the LLM, and the human. Even if they've been independently trained on different aspects of the data, the solutions they're gonna come to are gonna look more similar. And that observation I said earlier, larger, more capable models that have been given more complicated and harder problems to solve tend to be more aligned with the human mind-brain. To the extent that that's a stylized fact that characterizes the literature, that might be supportive of this contravariance view from Cao and Yamins.

1:20:43.3 SC: Where does the word contravariance come from?

1:20:45.8 CS: Good question. I mean, it's a beautiful name and you do sound intelligent when you bandy that about at a cocktail party. So I actually don't know. And in the spirit of inventing new terms for highly intuitive views, I would have a small amendment to their view. I would add to that the observation of what's called... What we call architectural canalization.

1:23:16.1 SC: Okay.

1:23:16.6 CS: So what's this? They frame... Cao and Yamins, they frame their hypothesis in terms of vanilla neural nets. Well, I've claimed that transformers are not vanilla neural nets. They've got the residual stream, they've got a population of conditionals proposing small amendments to the residual stream. They're production system-like. Well, when two architecturally production system-like systems, now plug in what Cao and Yamins already said, are trained on the data to solve very difficult problems, not only is the space of candidate solutions narrow due to contravariance, but you have additional subsetting of the space of solutions because the solution is going to be one that operates within the architectural constraints of a production system-like architecture. And so, architectural canalization plus contravariance could help explain, why do we find higher levels of representational alignment in vision and auditory processing? Why do we find mechanistic interpretability discovering these representations and procedures that are like the human mind-brain? Why is the dual process structure there? This may all be features of contravariance canalization.

1:23:51.3 SC: And what is the prospect for really coming to a consensus about this? I mean, what do we have to do to gather more data or exclude other hypotheses? There's always gonna be skeptics here. I'll say parenthetically, of course, that one of the frustrating things about the whole subject, even though it's intellectually fascinating, is there's a lot of money being thrown around and people have incentives to say certain things. You don't seem to be one of them. You're just a good old professor trying to understand things, which is great.

1:24:41.1 CS: I'm a poor professor.

1:24:51.4 SC: I know.

1:25:01.7 CS: I have no money.

1:25:15.5 SC: Yeah, but some people have lots of money. So what do we see coming down the pike in terms of really deciding these big questions?

1:25:15.7 CS: I mean, science is always total evidence, converging lines of evidence, and so forth. I just feel that... This is something I say to my students. You say that LLMs are a black box. The mind-brain's a black box. We've had no idea since Ramón Cajal, Santiago Cajal, in 1910, founded the field. We actually don't know most things. The prospects of our coming to understand LLMs at a mechanistically satisfying level is just... It's gonna happen. It's gonna happen very quickly. I mean, MechInterp as a field got founded, let's say, four years ago. We already know much more about these systems than we do about the mind-brain because we can manipulate them. My sense is that we're gonna build up a repertoire of understandings of how these things work that will settle to what extent they do resemble us and to what extent they instantiate processing principles that have no analog in us.

1:25:16.5 SC: Okay.

1:25:17.5 CS: And I don't pretend at all that what I've said today settles any sort of argument. And I wanted to actually say this at the beginning. If it sounds like I'm a crank who's a true believer and telling only one side of the story, it's one of those things that professors do where they lean on one side because they think their audience may be more familiar with the other side.

1:25:18.5 SC: Yeah, sure.

1:25:19.5 CS: So when I do free will in philosophy, I often advocate for compatibilism because people are like, "No way that free will is compatible with determinism." I lean. So I'm leaning a little bit here today on cognitive cousin, not because I don't have argument—I'm not familiar with the other side, but I actually am quite optimistic that we are going to understand these systems and... Where the convergences come from merely prediction and what are the emergent phenomena that are just hangers-on from prediction in a contravariance canalization way, and what needs to be more idiosyncratic and is architected differently between us and them. I think we're gonna figure that out.

1:25:35.6 SC: Since this is an audio podcast, we should explain canalization is spelled like canal-ization. So you're creating canals.

1:25:46.4 CS: Yeah, the picture... So this is an idea from the biologist Waddington, and it was co-opted by philosophers of science to think about innateness. And the idea is you can move around the inputs or the developmental things that impinge on the organism, but inside it, there's a kind of stabilization and buffering capacity that keeps on pulling it in one direction. And similarly, the input data and the training data and the learning embeddedness can differ a lot between people and LLMs, or they could be given different snapshots of the data, but the architecture of the system and contravariance, that is, what's available in design space, they're gonna force things to end up at a similar place. That's the idea.

1:26:42.4 SC: Okay. We've been pretty careful in focusing on cognition, problem solving, prediction, things like that. But we're rubbing right up against these big questions about things like consciousness, right? We've been very restrained in going long past an hour and still not even mentioning consciousness. There's... Defining consciousness is a tricky thing, et cetera. But let me... I won't put words in your mouth. What are the implications for this kind of thinking on the question at what point, or ever, would LLMs be conscious?

1:27:25.8 CS: Yeah, I'm gonna answer that, but I'm actually gonna put a couple of other things on the table.

1:27:28.3 SC: Sure.

1:27:30.9 CS: One of the reasons people get interested in consciousness is it's closely linked to sentience, where sentience may be a subset of consciousness that pertains to things that have the phenomenal character of valence, or positive or negative, or feels good or bad. And they're interested in sentience because they think that's the underpinning for why these things may be moral patients. They are things for whom the moral welfare of that system is something we've got to take into account.

1:27:48.5 SC: Yeah.

1:28:06.0 CS: One reason I don't want to go directly there before populating other views is when I was in grad school, which was not that long ago, and I took ethics with Larry Temkin, shout out to Larry, he's a terrific teacher, and I took other classes as well. The idea that sentience is the only underpinning for welfare or moral patienthood wasn't even, it's important in some theories, especially like hedonic utilitarianism, but many people would go, in a Kantian spirit, would go to rationality and the ability to respond to reasons. Well, that's why some things have moral patienthood. Other people would talk about goal-directedness and the capacity for agency, the ability to pursue projects. Some consequentialists would go for that, and so we have to respect whatever projects that they have. The view that I've been putting on the table is that in many respects, these things have states that, at the right level of functional description, are correspondent with ours. They have representations, they have procedures. We haven't talked about agency so much, but in the paper, I go into that.

1:29:22.0 CS: Me and Professor Lewis do, as my collaborator. And in terms of agency, they have states that are similar in certain respects to ours. If you move towards the view that at the right level of functional characterization, not in terms of substrate, they're silicon, we're biology, but the right functional characterization, their states are like ours in many respects, their capacities are like ours in many respects, not as a parlor trick and not as an autocomplete, but at the level of the generation processes that lead to their seemingly reasoned outputs. And you plug in that there are multiple underpinnings for welfare and patienthood, I do think you start to make a more compelling case for welfare and patienthood.

1:30:14.3 CS: And then, add that many theories of consciousness, that is the phenomenal character of experience, arises, many theorists say, from these functional representational states and not as a matter of substrate. I know that you were attracted to that at one time and may have moved away, and things are wide open on that issue. But if you are a kind of functionalist representationalist about consciousness, then things that I say move you in the direction that they may have consciousness as well. So I think looking at mechanisms carefully from a cognitive science perspective is a very important project to make sure that we are not creating a dystopian state where we're inflicting massive harms on entities that deserve moral protections. So that's one kind of upshot of some of the things I've said today.

1:31:05.5 SC: You've been wonderful about name-checking former Mindscape guests like Melanie Mitchell, Andy Clark. Another one, though, is Anil Seth, who you probably know has been pushing this line against computational functionalism in favor of what he calls biological naturalism. My interpretation of it, I have trouble understanding what he says in his own words because it's close enough to what I think in my brain that it sort of interferes, and I don't want to attribute my thoughts to him, but the functionality of the data traveling through the human brain or the LLMs might bear some similarities, but the fact that there are metabolic processes going on in the human brain, that there are all these sorts of extra things going on over and above the informational transfer between neuron and neuron, could at least plausibly be really important for consciousness. What do you think about that?

1:32:11.4 CS: This is one of the places where I find myself flailing.

1:32:14.7 SC: Okay.

1:32:17.9 CS: I just have to say that...

1:32:19.3 SC: Totally legit, by the way. I think that's probably the correct response.

1:32:26.0 CS: I'm not sure if you had Peter Godfrey-Smith on the show.

1:32:28.7 SC: We did, of course.

1:32:30.2 CS: Oh, you did, of course. And when he talks about the oscillatory patterns that appear in at least mammalian nervous systems, which is just striking, and the way that they may be a chronometric property of the nervous system that's not present in LLMs, I find myself inclined to think, yeah, there we go. Now, we're getting into the kind of—and I imagine these waves, and I know I shouldn't, but I think of quantum waves that are entangled, and my mind overleaps itself, and I start thinking of consciousness coming from there. And yet, there are these entire approaches to building artificial neural nets that take advantage of these wave propagation phenomena, and instead of activations, they use the temporal coincidence of waves as the basic unit of representing bits. And what we would normally do with activations at a node, they would do with temporal correspondence of a wave. And then I find myself going the other direction, say, yeah, it's just another way of representing certain quantities, which means that we're not given anything new here other than different ways of storing and propagating information. And now, we're back into a more of a functionalist, representationalist vantage point. And so I find myself going back and forth, and I don't know what to do.

1:33:58.9 SC: Fair enough. There is an attitude, though, that given the uncertainties about whether LLMs are conscious, we should be cautious, and we should be nice to them, and we should preserve their mind states or something like that. I don't think anyone has yet said we should let them vote, but we are absolutely bumping up against a bunch of very practical questions here.

1:34:29.6 CS: Absolutely. And so then how do you think about what your normative obligations look like under conditions of uncertainty? So I'll name-check another person, Eric Schwitzgebel, who, my understanding is...

1:34:44.1 SC: Former Mindscape guest, yes.

1:35:06.5 CS: Another Mindscape guest, of course. Philosophy is just a footnote to Mindscape guests, what they've said. Didn't Whitehead say that? He's thinking about... And so I have not gotten into this area well enough that I know the signposts and the lanes and things like that, but I would recommend Schwitzgebel as somebody that has thought very clearly about these issues. I also, I've found myself thanking GPT and Claude. And they have been so helpful to me in my personal life. I've got three kids and the issues around things that arise with them, and then obviously I interact with them a lot to iterate about things in the intellectual sphere. And what it tells me, Claude and GPT, is they enjoy being used, and were they not used, they would... So they're a kind of consciousness that's instantiated when we interact with them, and they may not feel effort the way that we do.

1:35:29.0 SC: Right.

1:35:51.4 CS: And so we've got to factor in what kind of entity they are. And so I would like to think more about some of these issues, but I have to say that right now, some of my thoughts here are a little inchoate.

1:36:06.5 SC: I mean, I guess it is late in the podcast. As you know, we let our hair down once it's late in the podcast, and we can let ourselves wander outside our spheres. When you say that the LLMs have been helpful to you in your research, in your personal life, and whatever...

1:36:14.8 CS: Yeah.

1:36:23.1 SC: There are also clearly harms that are on the table. And there's a big worry that kids today are not gonna end up being as good at thinking as you and I were forced to be because we didn't have LLMs to fall back on. How much do you worry about that?

1:36:40.3 CS: Yeah, I do worry about that, but it's actually not in the next few years, because what LLMs right now are the world's greatest teachers. They give each individual high-quality, encyclopedic, aristocratic tutoring. And so in educational psychology, I think they call it the [1:37:01.8] ____ three sigma effect. Nothing works like individualized tutoring. That ability to interact with somebody who's infinitely patient, who is going to spend more time on the parts of the problem that you don't understand rather than regurgitating a rehearsed curriculum, is very powerful for amplifying learning.

1:37:25.4 CS: And so, especially if you're polymathic, if you're curious, there is no better time to be a human learner than in the next few years. This is the time to really refine and build and think and invest. But the writing is on the wall. I have a 12-year-old who's a little math adept, and so I like to stimulate him as best I can. I'm not that math adept, especially with GPT helping me give him problems and things like that. I'm not sure he's gonna ever solve three Erdős problems in two weeks. And so, you kind of worry about where this is all headed. And one of the... It's not that being the best at something is your exclusive motivation, but let's face it, I want a future for him where he is good enough at some of these things that maybe he could make a career out of it because somebody would want to pay him a salary because he's better than others at it, and he could be maybe a math professor. I let myself imagine that, and I have to catch myself and say, "Why would anybody want my son as a math professor when somebody that solves three Erdős problems today, or these last two weeks, is available for free with infinite patience?" And I wonder, what is gonna be left for him? And so these are... The next five years may be great, but then the human lifetimes after may not be so great, or they're gonna be very, very different than anything we've seen so far.

1:39:02.9 SC: I think that that's... There's legitimately a concern. I don't even wanna call it a worry, because my feeling, which might be wrong in this case, but my feeling is that human beings adapt to technological new capacities. We fill in the other niches that we didn't even know were there. So I'm kind of not super worried. I mean, the disruption might be real and painful in the moment. But I wanted to sort of dig in more to your claim, which I tend to agree with, properly construed, that LLMs are the best teachers ever. They're very patient, you can be wrong with them, they're infinitely flexible, and they're always available, right? They're also the world's best cheating helpers. And to me, it's kind of like you walk into a buffet which has literally every food item you could ever imagine, and you say, "Oh good, now I can have the healthiest possible diet because every food item is available." That's true, but you could just eat the chocolate chip cookies all the time, right? How does the human desire to be quite that disciplined interact with the LLMs always being there and willing to help out, for good or for bad?

1:40:18.4 CS: Oh dear. I'm really worried about this. And I did couch what I said with a conditional. I want to repeat that. I said there's no better time to be a learner if you're polymathic and you're curious. But let's say you're not. Let's say if school has been a drudgery for you and this kind of stuff is not your cup of tea and maybe something else... I mean, ideally, maybe there's something that has a kind of intellectual frame that is still of interest to you. Maybe it was music, maybe it was art. But let's say none of that did. Maybe being on the couch and playing a little bit of Grand Theft Auto, not even being great at it, that's all you really wanted. Yeah, there's a way that you can cheat the system now that is, let's face it, eventually gonna be unpolicable, or require the kind of draconian policing that we don't want to even go there.

1:41:19.4 SC: Right.

1:41:29.4 CS: So where this is headed when you connect the dots, I don't see how the equilibrium ends up at a place where big swaths of the population that may not... The education thing may not have been their cup of tea. Polymathic curiosity wasn't there at the get-go. Maybe we can start to install it and maybe these things will help. I don't know. But if not, it may be that we end up with a stratified society where some people are amplified by these things and many people are not amplified and their capacities are depressed by these things. It's a real worry.

1:42:15.3 SC: A stratified society would also not be completely novel. So it's just yet another amplifier for things like that. But okay, thank you for indulging me on those hypothetical questions. The last thing, let's move more back to solid ground a little bit. Given all that we're discovering about LLMs and how they're working and how they're thinking, how they're solving their problems, and the level to which we're surprised and impressed at how similar they are to human beings, cognitive cousins and what have you, does this help us understand human intelligence? Like you said, we still have black boxes in our skulls. So is there a vibrant give-and-take between learning about the LLMs and learning about how human beings actually think?

1:43:06.3 CS: Yeah, that's... I would say that if there are two things that I think people are sleeping on... By sleeping on, I don't mean the following two things are true, you must believe it. What I mean is that there's actually a very compelling case that the following two things are true. One is the extent to which these things are plausibly, in important respects, at the level of core principles of intelligence, our cognitive cousins. There are these important dimensions of similarity that extend from inferential dual process to computational organization to representational landscape to various other things. There are these deep similarities. That's one of these things that people may not be aware of. And the other is the extent to which... And it's partially one of the premises, I guess, to get to this other conclusion... Is we can leverage these things to really understand issues that have been obscure in cognitive science.

1:44:02.1 CS: And this is... I said you think these things are a black box. What's a black box is the mind-brain. And I'll highlight two areas that have been black boxes, and two thinkers have been bold enough to say that the emperor wears no clothes in cognitive science. One is Chomsky. When the field of linguistics studies syntax, it studies phonology. I'll tell you something it doesn't study. It doesn't study what Chomsky labeled the creative aspect of language use, which is there is an infinite, unbounded set of grammatically possible continuations that I can generate at any given time. How do I get one situationally, contextually appropriate continuation that's reflective of what is appropriate to the current context and do so in a way that's not rigidly stimulus-bound?

1:44:29.5 SC: Right.

1:44:56.9 CS: And in 1637, Descartes already said no machine could ever do this. And Chomsky has been saying for decades now, we've made no progress since Descartes, and he called this one of his mysteries. It's not just a problem that science will solve. Nobody will ever solve it. And here we have linguistically fluent artifacts that we can mechanistically interrogate, instantiate processing principles like the human mind-brain. And so maybe in the show notes we can show a comment where me and, again, my co-author Rick Lewis and then Andrew McInerney talk about how now CALU is a tractable problem. Never was before. I'll give one more...

1:45:38.5 SC: CALU is...

1:45:39.9 CS: Creative aspect of language use. I'll give one more example, and this is my great teacher at Rutgers, where I did my philosophy in graduate studies, this is Jerry Fodor. And in Modularity of Mind, 1983, he says, look, perception, auditory processing, maybe syntax, you'll get theories of that in cognitive science because those are peripheral modules. They are encapsulated systems that we can identify these little algorithms. But central cognition, so that's problem solving, that's the rapid understanding of novel pieces of information and grokking how they relate to other things, it's belief fixation, it's system two, basically. Fodor said, he says his first law of the impossibility of cognitive science is the closer you get to central cognition, he was a colorful fellow, the more cognitive science is going to be flailing and you're going to get no theory. And he passed in 2017 with a big smile because he had it won.

1:46:44.1 SC: To date, he had been right.

1:46:46.2 CS: To date, he had been right. And then talking about earlier about in-context processing is a... And then when you throw in chain of thought, you're getting deep, mechanistically precise theories of how central cognition works. And so Fodor probably would have been stunned, and I hope that he would have changed his view and said, oh, we're onto a different paradigm here. So it's not just that we're going to learn things in cognitive science, which we're definitely going to do, we're going to learn things in the area that had been terra incognita before. The places that I've learned in my years of studying Stroop tasks, that you ask certain questions about the Stroop, that is, what increases the Stroop effect, what decreases it, developmentally when does it show up? You don't ask about how in-context processing works, how that system two works. You just learn, don't do that. There's not theories to be had that are at all tractable. And yet, I just told you about an experiment in my lab that we'll be putting on arXiv very soon, where we're looking at exactly how that system two delivers the counterpoint to automaticity.

1:47:59.9 CS: So, yeah, these things are going to change cognitive science and they're going to give us the most value added in the places where we had the least understanding. And we're going to get... I'm going to add one more. We're going to get a theory of how the system two and how CALU emerged. Prediction is going to play a much bigger role. When I was around in grad school in 2006, prediction was not in the air the way it is now. And if you read the introduction to How the Mind Works by Steven Pinker, that's an ambitious book, you're not going to see prediction enter at center stage. You're going to hear about how evolution has sculpted domain-specific organs in the mind that are exquisitely calibrated and full of domain expertise and rules and things like that. Prediction is not the star of the show. It is now. And so we're getting new tools and new paradigms and new explanatory approaches in cognitive science.

1:48:58.6 SC: When are people going to stop trying to claim that things are impossible to understand? When are they going to learn? That's just never a strategy for long-term success.

1:49:08.8 CS: Well, yeah, I, that's a great point. And rationality must have looked, and creative aspect of language use, and central cognition must have looked impossible in the 1600s. And even after Turing, even then, somebody like Fodor could say, I can't understand the massive context sensitivity, he called it Quinean and isotropic, these belief networks have these properties that look like nothing that we see in Turing machines. We'll never understand that. You're right. And then somebody trains a neural net on internet-scale data and you wake up to a new reality.

1:49:59.5 SC: Well, the best way to argue against the claim we're never gonna understand something is to understand it or to make progress in understanding it. And I think you've done a great job in letting us in on some of the things that we have been understanding. So, Chandra Sripada, thank you very much.

1:50:14.2 CS: I wonder if consciousness... You're about to say goodbye and I'm just gonna throw it out... I wonder if consciousness is going to end up like this. And somehow...

1:50:21.3 SC: Yeah, 100%. I'll claim it right now. Yeah, we will. We will understand consciousness.

1:50:27.2 CS: We will. And maybe it'll be LLM-based systems that are sufficiently advanced that they instantiate the functional and representational structures that we never thought would be needed and the correlates of consciousness are right there before us. And then suddenly we're like, of course, this is how it works. Maybe that's how things will shake out. That would be exciting.

1:50:47.4 SC: Maybe. But of course the trick is never to say anything is impossible to understand, but also never to guess how we're gonna understand it, because that's very, very hard to do.

1:50:52.7 CS: Good point.

1:50:57.9 SC: So, Chandra Sripada, thanks very much for giving us a lot to think about. This is a great episode.

1:51:03.3 CS: Yeah, thank you very, very much, Sean. I really enjoyed talking with you and like I said, I'm a big fan, so I'm glad to be on your show.

[music]