Creativity is one of those things that we all admire but struggle to define or make concrete. Music provides a useful laboratory in which to examine what creativity is all about — how do people become creative, what is happening in their brains during the creative process, and what kinds of creativity does the audience actually enjoy? David Rosen and Scott Miles are both neuroscientists and musicians who have been investigating this question from the perspective of both listeners and performers. They have been performing neuroscientific experiments to understand how the brain becomes creative, and founded Secret Chord Laboratories to develop software that will predict what kinds of music people will like.
Support Mindscape on Patreon.
David S. Rosen received his Ph.D. in cognitive psychology from Drexel University. He is currently a co-founder and the chief operations officer at Secret Chord Laboratories, a music-tech startup company. His interdisciplinary research program covers an array of topics: creative cognition, peak experiences, the neuroscience of music production and perception, psychedelics and STEAM education. David began playing the piano at the age of 8 and bass at age 15. He is the co-creator and bassist of sci-fi transmedia band, Chronicles of Sound, and instrumental progressive rock band, NAKAMA.
Scott Miles received his Ph.D. in neuroscience from Georgetown University. He is currently the CEO and innovation leader of Secret Chord Laboratories. He has been performing and producing music since the age of 10. In his doctoral work he investigated how music preference is formed in the brain. He secured funding through the National Science Foundation Graduate Research Fellowship to support this work. With David Rosen, Ph.D., he found support for two hypotheses about how the structure of music leads to purchase decisions. Miles then coded an algorithm to generate new music, and in a behavioral experiment, music featuring these properties was indeed preferred. He formed and has overseen the development of Secret Chord laboratories since it was incorporated in June 2018.
[accordion clicktoclose=”true”][accordion-item tag=”p” state=closed title=”Click to Show Episode Transcript”]Click above to close.
0:00:01 Sean Carroll: Hello, everybody. Welcome to the Mindscape Podcast. I’m your host Sean Carroll, and today we’re gonna be talking about creativity, what it means, and where it comes from. So I thought it would be… Well, it worked out anyway, that this would be a fun time to actually be creative in the format of the podcast. This is one of the very rare times when we’re going to have not two people on the podcast, me and somebody else, not even one person as I do in the solo podcasts, but three people on the podcast, me and two other people. We’ve done this only once before when we had Matthew Luczy on the program to talk about the pleasures of wine, and Jennifer Ouellette joined us as a guest commentator there. But now we’re gonna have two people who are experts in their field, and I will be talking to them. And that field is music and creativity.
0:00:49 SC: In some sense, creativity is mysterious for a lot of us. We know it when we see it, but we don’t know how to conjure it up. Music is a nice kind of laboratory to explore how creativity works, not just how we recognize creativity, but what is going on in our brains. So both David Rosen and Scott Miles are neuroscientists, and they are also both working musicians, so they’re interested in doing neuroscientific experiments to understand what’s going on in your brain when you’re being creative. They recently published a study of jazz musicians improvising on guitar, and they showed that certain parts of the brain light up when you’re improvising, and guess what? The parts of your brain that light up are different if you are an expert jazz musician versus if you’re just a beginner. So that might give us a little bit of insight into where creativity is coming from. Creativity is not the same for everyone.
0:01:44 SC: On the other side of the coin, the listener side, both David and Scott are interested in why do people like certain kinds of music, rather than other kinds? We like our music to be a little bit patterned, a little bit predictable, but not completely predictable, not just rote and repetitive. So they were actually part of the creative team behind Secret Chord Laboratories that is developing artificial intelligence software that will help you judge whether a certain piece of music is actually going to be enjoyed by the audience before the audience even gets to hear it. So there’s a lot of different aspects to creativity here, and we’re gonna talk about them in the context of music. But I hope that this actually extends beyond the simple musical realm to creativity more broadly. That’s the kind of thing that all of us could use a little bit more of. So let’s go.
[music]
0:02:52 SC: Alright. David Rosen and Scott Miles, welcome to the Mindscape Podcast.
0:02:56 David Rosen: Hi, Sean, thanks for having us.
0:02:57 Scott Miles: It’s good to be here.
0:02:58 SC: Let me just, because I’m not used to having two people as guests, let me help out the listeners. David, why don’t you say your name so we can recognize your voice going forward?
0:03:08 DR: Sure. Hi, this is David Rosen.
0:03:10 SC: Alright, that’s David, and Scott?
0:03:11 SM: Hi, this is Scott Miles.
0:03:14 SC: Excellent. And let me take on the devil’s advocate to what you folks are doing here, because you’re thinking about music and creativity, studying it scientifically, and even thinking about how we could algorithmatize… Turn our notion of what constitutes good music into an algorithm. Isn’t there some fear that you’re taking all the heart and soul out of a passionate human endeavor?
0:03:43 SM: Oh, well, there certainly is a fear. But I have full confidence that with the approach that we’ve been taking meticulously from the very beginning, that we’re doing everything we can to make sure that those fears aren’t realized. When you really look at it, what we… At the heart of what we do is we look at expectations. So the principles that we go into and all the science of it is that tension release and has to do with the expectations that are developed over time, and it has to do with culture, and culture is something that is a biological necessity of evolution. It’s something that we develop in the place where we are, so it helps bond us to each other, and it creates this… Helps mold this emotional bond that we have to help the cohesiveness of a territory.
0:04:36 SM: And so one of the functions of music throughout time has been to give an identity to a group of people who occupy a certain territory. And there’s also, of course, symbols or you have sports teams, are all these things that identify you. Your accent, your customs, how you greet each other. And these things are things that we learn through exposure. And so what we’ve tried to do is to model that sort of learning. That learning is statistical, and because of… Just it happens to be, and I think that this is actually not intentional, but it has benefited us through our evolution, that the place where we develop these statistical expectations through exposure is very proximal in the brain, in the limbic system, to our emotional centers, to the parts of our brain that process reward, and that process these… The dopamine and oxytocin and serotonin and these feelings of belonging, togetherness. And one of the functions that music has served in the past is to bond us together, and we’re so passionate because of this that it allows us to overcome the fear of, God forbid, we have to defend the territory. You see Braveheart, and you see them painted and their faces and that sort of thing, and then you go to a KISS concert and you see the same thing. And so you have… Or maybe an Eagles game, right?
0:06:03 SM: So, there’s something about this whole adhering to… This interplay between adhering to these expectations, which is really what culture is all about, and emotional… The emotional component to culture and music. And really, what we’ve just tried to do is look at that in an algorithmic way, you’re forming algorithms in your brain anyhow. And because of artificial intelligence, we have been able to scale that and say, “Okay, what might an individual audience… How might they respond to this?” and maybe help the industry a little bit that way. And let’s face it, the industry is already using algorithms, and the algorithms that they’re using are not human at all. We’re looking at something that’s a lot closer to the human experience, which is where the brain is processing the auditory signal.
0:06:56 SC: I think that’s a fascinating answer and actually very different than what I expected, which is great. Because, of course, I’m not very good at playing the devil’s advocate here. I think that we should be trying to understand what goes on in the brain when we do these things. But your emphasis on culture is really interesting. So I presume that when we think about how music impacts the brain, there are universal aspects that everyone responds to, things like rhythm and melody, but then there’s also culturally specific aspects, and it sounds like you spent a lot of time separating those out, or at least taking the culturally particular aspects very seriously.
0:07:31 DR: Yeah, I think that’s a really important part of this. One of the things we say is, “Neuroscience is people, and expectations is culture,” and what I mean by that, is that the brain is really what makes humans unique amongst all the species in this planet. Our ability to problem-solve and to think into the future, and to create plans, and organize, and create hierarchies out of information. And it’s the brain data, really, that differentiates, in terms of different analytics and predictive algorithms. The approach of Scott and I being two neuroscientists, how we want… How we think the most effective algorithms need to have that data point to be human-centered, and then… Go ahead, Scott.
0:08:20 SM: Yeah, and you’re right, Shawn. There are… There’s a little bit of… This is the classic debate in Psychology, and Neuropsychology, and Cognitive Science, that… About nurture versus nature. And there are many scholars who say, “Oh, the perfect fifth in the power cord, and music is universal and it’s in every system,” and when you look at the literature, you see like, “Oh, well, sad songs have minor thirds and happy songs have… It’s more consonant to have a major third. People prefer duple rhythm over triple rhythm,” but then you see, there’s some cultures… There was actually some work done by my colleague, Jessica Philips-Silver, with whom I worked at Georgetown, and she actually looked at babies who were… They gestated in… When their mother… You will develop a lot of your rhythmic expectations in the womb. So things that seemed to be universal might have to do actually with exposure to just the gait of their mother walking, and so, some of these things actually are environmental despite how universal they might seem.
0:09:35 SM: And there’s some work, I think it was Joshua McDermott at Harvard, he looked and he supposedly had found a tribe of people in South America somewhere, who did not prefer the consonance… Who do not find major thirds more consonance than minor thirds. But I actually had the opportunity over pizza, one time in Cambridge, to have a debate with him over this, and what I said was, “You found a lack of results, but you didn’t find opposite results. It would be a lot more interesting… ” not to say that it wasn’t a great work, but I would think, if I were doing the study, I would say, “Oh, but it would be more interesting if you found somebody who had the opposite effect.” and we don’t find that. And so, that might mean that you’re right, Shawn, that there are some things that are universal. And what we try to do in our algorithm is, we have… There’s a lot of layers that we build into the machine learning and the artificial intelligence, and it has to do with exposure, and some of it is just baked into the cake that these things like major thirds or perfect fifths or duple rhythm even, for most cultures, is something… I mean by four fours instead of three four, is something that is preferred, but it’s a controversy.
0:10:55 SC: I mean, even the 12-tone scale, the chromatic scale, and then the major scale, and minor scale within that, those are not culturally universal. Other Asian cultures and African cultures don’t always choose that as the… Am I correct about that?
0:11:10 DR: Mm-hmm. You are.
0:11:10 SM: Yeah, actually, my initial PhD study idea for my dissertation was going to be, to see if grown adults, undergrads, could learn the different tonality and the different syntactical rules of North Indian classical music over a few exposures and see what’s happening in their brain in FMRI, in Functional Magnetic Resonance Imaging, as these rules were being learned. And there’s some foundation for that, from an artificial musical system, it’s called Bohlen-Pierce. Which is kind of… To me, it’s kind of crazy, but it’s weird sounding, it’s from the 60s, it was experimental… I think it breaks the octave into 13 tones, instead of 12. So, what we wanted to do with that work, and it kinda goes into what our whole idea of going into the secret chord, is that, we wanted to do something that’s more applicable and more generalizable to what people are actually doing in their everyday life.
0:12:20 SC: Cool. I think… I have a million questions, so I’m hesitating, and therefore, I think what I really should do is, back up a little bit. Lets… This is exciting stuff, so lets lay some of the groundwork for how we’re thinking about these, and let me just go back to the most basic question. How do you two define music, what is music, what does it… Why do we all, throughout every culture, care about it? Which I guess, begs the question, are there cultures that don’t have music at all?
0:12:48 SM: Yeah, so that’s actually one of the sticking points in the study about the consonant third. That culture didn’t seem to have music in any sort of significant way.
0:12:58 SC: Wow.
0:13:00 SM: So that’s why I was… That was my counterpoint in the Pitzah discussion was, it’d be interesting if they had a completely different kind of music, but it’s just an absence of music. But really, we find, in one of the famous landmark studies, that we base a lot of our research on, out of Robert Zatorre’s lab in McGill University of Montreal, was… The lead author on that was Valorie Salimpoor, and she starts off the paper, in her science paper, in this experiment. It was an empirical experiment, but to start the literature review, she brings up that there was a flute discovered from, I believe it was 10,000 years ago in Germany, made out of bone. And so at the very dawn of, really what we consider any kind of human species that we would recognize as far as civilization, there was already music.
0:13:53 SC: Huh. And what is the definition? Maybe we’ll give this one to David, like how would you define what music is?
0:14:00 DR: Hmm. I would say music’s a nonverbal form of communication that involves a series of different… Can be broken down into a series of different features and aspects. And music can have any of these aspects, it can have a few of these aspects. Dynamics, I think, is an important part, and melody, melodic, so in terms of the movement and pitch of notes and how that shifts. I think rhythm is an important aspect of that. And rhythm doesn’t just show up in music, of course. We have rhythm to our speech as well. I also think that the texture of the sounds is a critical piece, and that goes right hand in hand with the timbre, which again, we see a lot of overlap in terms of this… That’s why I think the nonverbal communication. But I wouldn’t say it’s not a language, though, because clearly there’s linguistic components in music as well. So I think those are some of the core components of it. I don’t know if that’s a clear definition or operationalized. Maybe Scott can give that in a more of an operational definition a shot.
0:15:08 SM: Well, it was interesting, I don’t know if… It must’ve been… Something tells me it might have been a drummer, but there was a reductionist kind of definition of music, that he said everything, when it comes down to it, is just rhythm. Because if you play something fast enough it becomes… If you play a beat 440 times every second, it becomes middle A, or high A, on a piano. So… And then you play overtones over that and then it becomes a timbre, and the one thing that you left out there I think was harmony, which is interesting as an interesting omission, because when I looked at the North Indian classical music, they have no notion of harmony. It’s almost exclusively monotonic music, so… At least the music that I encountered.
0:15:52 SC: I remember hearing somewhere this claim that that sort of more primitive, what we would call primitive music, ancient music, but also music from other cultures that we Western-centric people might not grow up with, has actually much more sophisticated rhythmic structure than Western classical music does, and when Western classical music discovered polyphony and harmony and so forth, their rhythmic capabilities atrophied a little bit, and it’s more work these days to actually rediscover that heritage.
0:16:28 SM: Well, David Byrne is an interesting academic source there, he’s good friends… From the Talking Heads. He’s good friends with someone who’s actually advised us along the way, Daniel Levitin, who’s also at McGill, who’s probably the scholar to ask these sort of questions, the foundational questions of why we have music and that sort of thing. He’s expounded upon that in book upon book. But in David Byrne’s book “How Music Works,” He said… The first chapter, or the first couple chapters, is all about environment, and he talked about CBGB’s and how the structure and the architecture and the wall hangings and everything just shaped the sound of the Talking Heads early on. And he talked about how Mozart and the Protestant Reformation, why that started. You had more intricate sounds when you had droning sounds in the cathedrals because they would resonate through the walls. And he talks about Africa and South America, and then when you’re outside, what’s going to travel far is going to be rhythm, it’s gonna be the bass, the lower notes in the frequency. And I don’t know if that accounts for any of the phenomen0n that you’re bringing up, Sean, but it’s something to think absolutely.
0:17:39 SC: Yeah, yeah, absolutely. Okay, so, speaking of rhythm, one of the things that you’ve done research on is the importance of surprise or the role of surprise, let’s say, in how much we like music. Like if all we had was a rhythm that repeated itself in a very basic way over and over again, we would get bored pretty quickly. Whereas, if all we had was either rhythm or melody, for that matter, that was completely random and unstructured, we would be repulsed. There’s some happy medium there, and I know that you’ve been interested in studying the extent to which a little bit of familiarity and a little bit of surprise plays an important role in how much we like a piece of music.
0:18:19 DR: Yeah, so we definitely are looking at expectations being violated and surprise. And there’s these two aspects, right, there’s the familiar… There’s the familiar and the surprising. And there’s been work done on both of these topics, but what we really find interesting and what we found through our research is that when you have this right balance of familiarity and surprise, that’s when you see the optimal level of enjoyment and dopamine, and really pleasure. And where we see this, why this is occurring, the mechanism in the brain is really because of our two memory systems. We have an explicit memory system that is constantly learning, it’s a conscious system, it’s our… If we relate it to a language model, it’s declarative. In terms of our memory system, it’s declarative, we can recall it. It’s when we’ve heard a song over and over again and we can sing along with it, right, and we’re familiar with that. But as you’re saying, if we heard the same rhythm repeating over and over and over again, that’s not going to be pleasing.
0:19:26 DR: And it’s our other system in the human brain that responds, we’ve seen through a number of studies, like from Ani Patel of the implicit system responding to surprises, to chords and melodies that are unexpected. And basically when the memory… When both memory systems, when music is occurring and both memory systems at the same time are being hijacked by the musical stimulus around us, that’s when we see that dopamine response and when people are really preferring and finding their favorite types of music.
0:19:57 SM: Yeah, so it seems… So, like a lot of things, when you look at a person, they look like a single person, but when you study neuroscience, you realize there are these different systems at play. And Amos Tversky famously won the Nobel Prize for their work, Kahneman and Tversky, in economics, for looking at system one, system two. And my mentor at Georgetown looked at this with language, with procedural memory and declarative memory…
0:20:24 SM: Sorry, you might as well… I think you should probably explain more what system one, system two are, ’cause I’m sure we’re gonna get back to them.
0:20:29 SC: Sure. So it’s kind of… Yeah, yeah. So it’s kind of what Dave was talking about before. So your system one is the more implicit, the more like rules things that… If you drive home and you forget that you drove home because it’s something that you’ve done over and over and over again, and it’s something that’s automatized in your brain, but then all of a sudden, if you’re driving and then you realize, “Oh, I was supposed to go to the dentist and pick up my kid on the way home,” then that’s more a system two, explicit. And a lot of Dave’s early work… Dave’s PhD work had to do with these two systems in jazz improvisation, but just a…
0:21:08 DR: Scott has a great story about seeing Michael Jordan in 1992 that I think really clearly articulates the differences between system one and system two.
0:21:17 SM: Yeah, so I think that, yeah, so I’m not a big sports guy, and so even though I led with the Eagles reference, I pretty… When I think of the Eagles, I’m more likely to think of Don Henley and Glenn Frey, but… But, yeah, so, whenever I go to a sports game, I’m like, “Which one’s the bass player?” And so… But then I went to see, because I was dating this girl and she had tickets to see the Bulls and the Bullets in 1992, so it’s like the dream team or whatever it is, the Chicago Bulls in the early 90s. And I went there and I had never been to a professional sports game before, and I’m sitting there, I’m just trying to get some popcorn or whatever, hang out with my girlfriend and her father. And I look out there and I’m like, “Wait a minute, one of these guys… ” There’s like 10 guys out there running around, and nine of them are doing what I’ve seen on TV or whatever I’ve seen in the playground and all that stuff. And then this other guy is performing ballet, and that was Michael Jordan. And I was like…
0:22:19 DR: Just a different level.
0:22:20 SM: Yeah, there was this emergent property of the way he was moving through the court that was just qualitatively distinct to me, and I had no idea anything about basketball, but I knew that that must have been the guy.
0:22:34 DR: Right, so…
0:22:35 SC: Okay, sorry. So tell us how this explains system one and system two.
0:22:38 DR: Sure, so I can dig in to that. So, because in Scott’s situation, he really lacked the… System two would be the system that’s focused on more explicit, controlled, and conscious knowledge. In terms of neuroscience, we think of that as the executive control network if we’re talking about brain systems. And if I’m a fan of sports and basketball, I know all about how many championships Michael Jordan has won. I know about the slam dunk contests he’s won, I know where he went to college, I follow him, I have his jersey, right? I have all these different reminders that have built up that my knowledge set of Michael Jordan, right? And let’s say I’m a huge Michael Jordan fan because of this, of all these stats that are true, explicit knowledge, with the fact that when Scott is watching it, he’s really having a more system one dominant experience of watching Michael Jordan, because he doesn’t have all of that meta-information. He’s responding to his movement and his visual… The visual stimulus that he’s having at that game, probably also with some cheers around them, with some auditory stimuli as well, but that’s more of an implicit bottom-up type of processing. It affects the dorsal stream, and in terms of neuro-scientific systems, we think more along the lines of the default mode network.
0:23:50 SM: Yeah. So to wrap up the reason that people, when they see that, those two things in concert are working, what it seems to be that they’re working against each other, but what you really want is you want something that’s very familiar explicitly, but very surprising implicitly, and it makes sense to me to think that it’s like… Dopamine as a learning molecule, so if you know something, it’s like if you’re watching a movie, like I think it’s safe to say Usual Suspects… If nobody’s seen it, spoiler alert. So once you get to the end, if you’re watching that movie with a friend of yours, and he or she hasn’t seen it before, when you get to the end, you know explicitly that there’s a twist there, but you actually enjoy it even more because you’re smarter than your friend, and your dopamine is going off and saying “you know something,” and it tricks you into thinking that you’ve actually done work and that you’re more evolutionarily adapted to your environment.
0:24:49 SM: And if there’s nothing there… Like you said, Sean, if it’s just going on and on and on, if it’s just like nobody… They never find Keyser Soze, and that’s just the regular plot and there’s no twist, there’s nothing to learn, there’s nothing for you to know. And conversely, if you don’t know, then there’s nothing for you to be smart about. You’re not smart about the thing that is there. So that’s the reason that you want new artists, but then you wanna play the grooves out and spin them on the radio or streaming, because you want to have that fresh sound, but you want everybody to hear it over and over and over again. And it seems like those two things are diametrically opposed, but the missing premise is there are two different brain systems, and that’s what we’re seeking to… That’s what our laboratory work is seeking to prove.
0:25:37 SC: There’s a joke here in Hollywood, when you go to pitch your TV show to a network, the network will always say, “Give us exactly what’s already successful and on the air, but completely different.” And maybe you’re explaining that. Maybe this is not just hypocrisy, maybe that there is this… Once you understand the dual process notion of how the brain works, there is… At one system, you wanna be exactly what people know, and in another system, you wanna be surprising them.
0:26:04 SM: Right. Well, don’t they ask you for a “meets”? Like you have to say, “It’s Alf meets Terms of Endearment.”
0:26:10 SC: That’s really helpful, right, yeah.
0:26:12 SM: Yeah, and that would be a great movie, by the way. I think.
0:26:17 SC: I wonder how formal we can be, about extending this analysis to things like movies or novels or something like that, because in some sense, music is a nice simple test case, songs are short, there’s a finite number of instruments, etcetera, we can analyze the notes, but maybe this is a more general theory of how we respond to art, maybe even visual art as well.
0:26:44 DR: Yeah, yeah. I think it definitely applies. I think your inclination is correct. Sean, we talk about this all the time, about narratives and story, I’m a big sci-fi and fantasy nerd and I always compare it to an epic trilogy, it’s like the same thing, the second act, where you’re building attention, you’re building attention, you end on a down moment, and then that’s the tension similar to a musical moment, you have that release of a huge epic battle in the third book, and that’s the… That’s that release, and that’s where the pleasure really kicks off, is that… Those points between that, the push and pull of those stressors and then releasing it, which I think, ultimately, is rooted in our evolution in which we experience it through that.
0:27:27 SM: But the practical part of that, Sean, is, I think you kinda put your finger on it, is that, with… We have seven features that we look at in music and it’s very… Music is… Lends itself to being analyzed quantitatively, and because the Spotifys or the Pandoras and the academics of the world look at features, for other reasons, for song similarity and recommendations, we’ve piggy-backed on a lot of that work. But when you’re talking about… Just like if you reduce one frame of a 24 frames-per-second movie that’s in 4K, you’re talking about 8 million pixels, times 24 frames, and then you have all the different qualities and then you have… How do you generalize and make fuzzy boundaries around it, and it gets a little complex, but that there’s always work on the horizon to be done.
0:28:18 SC: Well, no one ever said that a complete scientific theory of the success of art was gonna be easy, so I’m glad that we’re starting with a lot of energy here.
0:28:26 SM: My mentor actually started… My mentor that I finished out the PhD with, works with renaissance paintings, and that’s sort of thing… With the visual neuroscience system, he worked with Tomaso Poggio, a famous visionary and vision at MIT, and so… And he’s actually worked in neuroaesthetics with that Norberto Grzywacz, so…
0:28:51 SC: But there’s a relationship here, reminds me of work that was done trying to claim that abstract expressionist paintings like Jackson Pollock were in part, successful, because they were scale-free, that is to say that there were things going on, on very tiny scales and things going on, on medium scales and things going on, on large scales, there were fractals. Yeah, it was debunked a little bit, because the methodology was not that great, but there’s clearly something to it, and I wonder how that also goes into the music thing, but surprise is one aspect, but also, this complexity is another aspect.
0:29:24 SM: Sure. Exactly. And I would also argue that there’s familiarity in the fact that you know it’s a Jackson Pollock. If somebody gave you the same painting and said their kid did it, then…
0:29:36 SC: Well, I think that the hope was that… Yeah, we’ve all had this reaction, “Yeah, I could do that. My kid could do that,” but the idea was that, in the paintings that had become historically very successful in abstract expressionism, there was something that was not in what your six-year-old does, it really was a quantitative difference, but I think it turns out to be hard to measure that. But you think… When you think of something like that… Have you studied whether something like that is also true in music, that it’s not just the surprise aspect, but multi-scale aspect, is that an important thing to people enjoying things?
0:30:11 DR: Yeah, I think that’s… I think this is expectation, violation, and surprise, has really been at the heart of what we do, where it’s a small piece of really, the greater phenomenon of how music moves humans on a greater scale. One of the other things that we’ve talked about from… Work from Carol Lynne Krumhansl, called Cascading Reminiscence Bumps, and that has to do with the influence and seeing how music comes… Popular music especially, comes back every 20 or 25 years or so, you see the 70… You see the 70s creeping back in, and the 90s, and then, you see the 80s, they come back, and these are cycles, because of when we’re younger, really, the music around us is not self-selected, this is during that critical period, where our implicit systems, those unconscious responses to the statistical regularities and characteristics of music are developed, and that’s really based on the decisions of the music our parents and our family, our family members choose. And then, so, we…
0:31:13 DR: That kind of sets this implicit standard, and also from which the next generation of music comes from, and the listeners who were growing up, they’re ready for this certain amount of expectation, violation, and surprise, and I think that’s really interesting to think about, because when those huge identity moments of the teenage years, those years when you’re really forming your social bonds and relationships, I think that’s where we started with talking about power of music and how it’s so important there. It’s interesting that, in terms of popular music, you really see the 13 to 19 or 20-year-olds, they’re the ones who are consuming and who are target of the most popular music, and it may be that… And we see this consistent trend of those top songs having these patterns of expectation, violation and surprise, because they’re at a set point of… They’ve learned the rules and what’s going on around them in the universe, and now, they’re ready to take that to the next step, and we see these cyclical patterns over decades.
0:32:12 SM: And to tie in the cyclical patterns to your previous point, Sean, about the Jackson Pollock, and the different levels of complexity, I think that it’s important even in our work, even if we do look at surprise and even if we’re looking at the same seven features, that you take into account that, some of the features can be looked at, not just as the dependent variables, but kind of to be looked at to say, “Okay, these… The values that we find for these parameters, whether it be rhythm or whether it be the year of release or whatever it is, certain timbres, they can actually set the stage and say, Okay, it’s not a linear effect, so I could say This is the condition upon which we say, Okay, this is now what we’re… This is the paradigm that we’re looking in. And it almost shifts to a different thing that wouldn’t be expected because of things like, Oh, well, their parents were listening to this music. We’ve got to incorporate that in the algorithm. Or look, there are certain levels of complexity, when you brought up the Jackson Pollock, it makes me think of John Coltrane. I’m a big jazz fan. My last name is miles, I named my son Coltrane. Coltrane Miles.
0:33:23 SC: Oh wow.
0:33:24 SM: And so giant steps. I’ve seen… I don’t know if this is real or academically rigorous, but I kinda hear it myself, and I don’t know if it’s wishful thinking, but I hear sort of fractal patterns in that composition of Giant Steps, and it’s almost like he’s composing the composition around the composition. And it’s like it turns into this kind of fractal sort of… I don’t know how much rigor, mathematical rigor, this would withstand. Like you said, just the same thing as Jackson Pollock, but that kind of thing is not something that linearly follows from a pop song. You have to go to a completely different paradigm and say, Okay, Coltrane fans are listening for something completely different. They are ready for something else. They’re not gonna skip after 30 seconds if there’s not a hook. And so you’re looking for… You kinda have to shift your expectations in making the prediction.
0:34:22 SC: Well, it’s really interesting how there are these universal aspects, but they play out differently in different people, depending on their culture, their training, their interest, etcetera, but speaking of that, the mind-scape listeners are the Coltrane fans of the podcast world. We went a little bit… We’re ready for a little bit more depth here. And I really wanted you to explain a little bit how you measure the surprising-ness of a pop song or any piece of music, because I know information theory is involved, I know we had a wonderful podcast with Karl Friston, who has a theory about how the brain works that is deeply embedded in information theory. So that would be a great connection.
0:35:01 SM: Yeah, that was a great session, I really got a lot out of that. He’s singing our tune.
0:35:08 SC: Yeah, exactly.
0:35:09 SM: Karl Friston. So yeah, the calculations basically… So we started off with… Talk about Giant Steps, we started off in baby steps, right? So we basically looked at zeroth order, so in other words, zeroth order harmony is what the dissertation work was on when I was at Georgetown, when I started working with Dave as a research partner. So we basically looked at a few hundred, like 600 songs, all in a major key, all from Johnny B. Goode to smells like teen spirit, 1958 to 1991. A little sampling. Yeah, so of the billboard charts and it was transcriptions that were done by a PhD student at McGill, Ashley Burgoyne. And so basically, we had all the pieces together to work from. And the theory, which was developed by Leonard Meyer in 1956-57, which is that information theory has something to do with meaningfulness. He didn’t necessarily talk about preference, but we kind of morphed it into our measure of Billboard charting success, which has its own confounds, and everybody always has a complaint about that, but the more we go to direct behavioral studies and with actual participants, then you can stop worrying about the confounds of the billboard methodology. But that’s what we had.
0:36:36 SM: And so basically, we looked at the top quartile, which is mainly number one, maybe some number two songs, ’cause of the long tale of popular music, and then the bottom quartile which is like number 50, 60, whatever, down to 100. And so what we did was we would take using the Shannon entropy, Shannon calculation from 1958, I believe, of Claude Shannon, and his way of computing complexity or entropy or information in anything that’s mathematically or anything that’s really classified into the different groups, and that you have a whole corpus of it, so we had the corpus, which was the 600 songs that translated to about 30,000 chords. And we took the chords and we cut them off at a 7th, so a 7th and a 9th and a 13th are pretty much the same. We figure the ear doesn’t really discriminate that much, and what I’ve learned from my jazz teachers, you can always turn a seventh into a ninth if you want. So depending on how jazzy you wanna be, right. So that was kinda… We’ve cut them off at a seventh, so basically there was about 500 or 600 different types of chords, all in one key. So we normalized everything to like a C major or whatever it is, whatever… One…
0:37:50 SM: So we’re differentiated between different keys, so it was mainly a matter of the function of the key and then the color of the… The function of the chord and the color of the chord. And then we would take… Okay, so let’s say it’s like one major, C major and C, and so that was the most common, I think you would probably guess if you had to know anything about music, popular music, that was the most common out of the 30,000 chords that are in all these songs. Not just each unique chord, but every time there’s a chord in all the songs, we put it all together. And we would say, Okay, maybe there’s 5,000 times that there’s a C major. If I break all the songs into quarter notes and I take 5,000 divided by 30,000. You get that number, it’s a fraction, you take the negative log base two of it, and so what you get is you get a number in bits of how surprising that chord is.
0:38:44 SM: So that chord is not very surprising at all, it’s low, around one or two bits, and so our range was between one or two bits to six bits, seven bits. If there’s a F sharp minor ninth or whatever, 7th, whatever, that would be seven bits. So it only showed up once out of 30,000 chords. So that gave us kind of a manageable scale to work with where we could see by taking the log, and Claude Shannon obviously came up with that method. And so that gave us a way of saying in bits, how much information is there from each one of these chords. Now, if you know anything about music theory, you might be listening to this method and saying, That’s ridiculous.
0:39:26 SM: If I hear a C-major chord in the middle of something, like it might be very surprising. I don’t care what… But we were basically just looking at zeroth-order and that’s it. So it’s very dumb… The algorithm itself is dumb, it doesn’t know what came before it, what came after, what context it’s in. It just knows what key it’s in and how often it happens. And we just basically looked there and surprisingly, we found something. What we were able to find was that… The first finding was that there was absolute surprise, that was higher, just barely like half a bit higher in the top quartile songs than the bottom quartile songs. And that was consistent across the whole song. So no matter where we looked in the song. But we weren’t lining them up in sections. Then we kind of came up with… The more we…
0:40:17 SC: Just to clarify, ’cause that was a great explanation, but there was a lot there. Let me just repeat it in my own words, so you’re saying that the most popular songs were ones in which you were slightly more likely to hear a chord that was slightly less popular than in the less popular songs.
0:40:33 SM: That’s exactly correct, yes.
0:40:36 SC: Okay, that’s an interesting finding, but now… But then there’s also within the song, you were getting into…
0:40:41 SM: Right, so in order to do that… So Dave and I came up with the idea that we were gonna look and line up the choruses and the verses. And so… It took some convincing because I was dealing with someone who was used to looking at paintings and visual, that sort of thing, and didn’t know a lot about, wasn’t an expert in the musical field. But Dave and I were pretty sure that that had to be something to do with like pre-choruses and choruses. And so if you know anything about Max Martin, who’s behind… The writer behind and producer behind the Backstreet Boys and Britney Spears all the way up to Taylor Swift. And so he… He always has these really massively tense pre-choruses and then boom, drops the chorus right in your face and then… And he lives in a wonderful, amazing mansion in East Hollywood, so…
0:41:34 SC: Got it.
0:41:34 SM: In West Hollywood sorry. So… We visited when we were out there, so I just wanted to see it. [chuckle] So it works. So we were trying to see, “Okay, is this something that just works for him or is this something that you can actually empirically show is a correlation.” And what we do is we lined up… We basically just took… Thankfully, Ashley, as parts of his dissertation, he had the transcribers that he used actually say what they thought was the chorus and the verse and that sort of thing. So we looked for all the verses that came right before a chorus, and sure enough, there was higher surprise… The surprise effect that you found over the whole song, was being accounted for almost entirely by the parts before the choruses? It was like… And the bridges… Obviously, but bridges came before courses too. So it was really something that looked like it was across the whole song, but it was because we were agnostic to where in the song, obviously different songs would have different places, we were just using like each song as one, and we were just saying like a halfway through the song, whatever, it’s still… A quarter way through the song. And it just happened to be that all these were kind of distributing themselves across all the songs, across 100 songs… 500 songs.
0:42:52 SC: So we have expectations for certain kinds of chords in general, but also at particular points in the song, we have expectations for what’s gonna happen next, and just tweaking that expectation a little bit helps us make a hit single.
0:43:07 DR: Yeah, it definitely helps to… Using those expectations… It helps in terms of how much of… Let’s say, commercial success is being driven by these expectations and surprise. And that’s a question that we ask ourselves… That we ask ourselves every day. But I think it speaks also to the common structure of popular music and how that’s something when… People who are not musicians or musicologists are familiar with that the sing-along the chorus the more wordy verses with more kind of… That’s another thing that we kind of talk about is how do rhythmic patterns of words in music also also have impacts or of lyrics. So I think that, again, speaks to the explicit system and knowledge and know-how of how these pieces of the song work, but then also when we looked before and after these key transition points is where we see these large transition points and changes in those surprise moments.
0:44:11 SM: Yeah, and what was fascinating to me is the chorus is the part everybody knows, it has the name of the song in it usually, it’s the loudest part, it is the hookiest part, but at least by the metric we were looking at, and that’s a big qualification, harmonic surprise, at least by that difference… There was no difference whatsoever. The choruses looked identical in the bottom quartile songs and the number one songs, they looked identical. So what I tried to say was, it’s almost like when you go on vacation. If Dave and I go on vacation and we go to the same place or whatever, we go to the Blue Ridge Mountains for a weekend, and Dave worked all week, I’m gonna make up a crazy, unrealistic situation, if Dave works hard all week and I didn’t do any work, I don’t know if that would ever happen but he would enjoy… Let’s say he would enjoy the vacation more because it’s more of a… It’s the same vacation, it’s the same chorus… It’s like my dad says, he says the joke… I told him about this finding, he says, Oh, it’s like the guy who they asked him, “Why are you beating your head against the wall?” and he says, “Because it feels so great when I stop.”
0:45:18 SC: Right. Right. Well this is… This is what I was…
0:45:20 SM: And it has to do with an actual physiological thing, which is you build up these opioids in your brain that are responding to the pain and that chorus, that verse can actually… It’s been shown that that actually causes this minor painful reaction in your brain.
0:45:36 SC: Well, this is what I was wondering about because you can say that a little bit of surprise is good, and that’s probably clearly true because like we say, “We don’t wanna just do the same thing over and over again.” but maybe not all surprises are created equal. Like you say, are there hidden variables here, are there other dimensions along which we say, “Oh, this kind of surprise makes a song especially compelling, whereas, this one would just be annoying to us.”
0:46:00 DR: Sure. So that’s… It’s again something we think about and look at quantitatively in our analysis. And so far, we’ve been speaking about harmonic surprise. There’s also surprise in other aspects of the music. There’s… In terms of the rhythm or the melody or the dynamics. But I think we can really understand how surprise and uncertainty is related to pleasure and preference. We think about an inverted-U, Berlyne’s inverted-U, the complexity of a stimulus, and how that’s related to pleasure. And ultimately that… If we think about the x-axis as complexity, we can imagine that any single person could have that inverted-U shifted along that axis based on their previous experiences in their life. If we go back to the fan of avant-garde free jazz, you can imagine what their set of expectations are and how and where on the complexity axis they fall is going to vary based on… By the individual.
0:47:07 DR: So I think a key factor… Because in that example, we say, “Okay, avant-garde jazz, I’m ready to lock into a 30-minute improvisation with very little noticeable structure and even harmonies that seem like they don’t really make sense and unclear chord structure.” But that’s it. It’s having that context and knowing and waiting for this. I’m a huge improvisation guy, that’s where my side of the work comes on as a fan of music, improvisation, and an improviser for 20 years in different bands. And for me, I live for the moment, that’s 15-20 minutes in, where you find this unique musical place. Where maybe you even trudge through some not very moving or even good music. But then you reach this epiphany together and almost let go off of those evaluations and judgements and really sink into this new place that you’ve reached. Which you wouldn’t have gotten to if not for dealing with those kinds of… Those moments that maybe weren’t as good.
0:48:09 SM: Yeah. So there’s a sweet spot, and that kinda gets to where we are now. After we’ve left, we’ve come a long way since those baby steps. Now we’re looking at things, like Dave said, not just harmony but melody, rhythm, timbre, texture, dynamics. And so, these kind of things we’re able to glean through music information retrieval automatically at scale from thousands of songs at a time from audio signal. And so that’s one of the things that we’re able to do that we didn’t really do. We needed this kind of PhD project from somebody else, we’re building off of when we did the work there. But now as a company, Secret Chord Laboratories, what we’re able to do is scale this and look at all those different things.
0:48:52 SM: And when you talk about the sweet spot, what we’re able to do with artificial intelligence, which is kind of where all this kind of feeds into the signal chain. All this information goes into our engine, a dopr. And people go over to dopr.com, eventually, when we have the engine completely up and running, we’re doing beta test now, ut they’ll be able to… The engine will be able to say, “Okay. In the past, in this context… ” with these kind of predictions that you’re trying to make, whether it be free form and improvisational jazz or whether it be Max Martin songs from the 90s or whatever it is. This is the kind of weighting of the systems, or these are the sweet spots that the algorithm and the artificial intelligence through the machine learning will find and apply to get the best predictions that show… That account for the variance in the past, and then apply those to the new songs that you’re uploading.
0:49:50 SC: Well, let me ask more about that process of creating this improvisation or even just creating music in general. I know that you’ve done a little bit of research on the neuroscience of this. The way that the brain comes up with a piece of music, either sitting in your studio, thinking about it very hard or literally there on the bandstand in the moment, improvising along with people. What’s up with that? That always seems profoundly mysterious to me. Are we beginning to learn more from neuroscience about what’s actually going on?
0:50:21 DR: Yeah. So that’s… That really is a central issue of my research. It’s been using jazz improvisation as one form of musical creativity, and what happens in the brain when people are generating, whether it’s an improvised solo or they’re recording multiple takes for a vocal take for a chorus. Luckily, I was able to look at this intersection.
0:50:48 DR: I was at Drexel University. So I was kind of looking at… I had two really great advisors over there, John Kounios of the creativity research lab, who had done decades of work on insight and the Eureka moments in creativity. He was my psychology advisor. And then I had an engineering and music information retrieval professor, Youngmoo Kim, at the Music and Entertainment Technology Lab. And in that lab, he had been… He was an MIT graduate and came from the Media Lab. And when I started grad school, he was… He had a lab of engineers and they’re basically like a feeder for places like Spotify, and Pandora, and Graystone. And he saw the importance, about 10 years ago, of not just looking at the engineering side but really bringing in someone with a music psychology background in cognitive neuroscience. Because by leveraging both of these kinds of tools, we could have a rigorous investigation of what’s happening in the music and what’s happening in the brain as people are being creative.
0:51:54 DR: And so through my work, I did a series of studies that looked at both jazz pianists and jazz guitarists, and correlational and causational studies that showed really distinct brain patterns between experts and novices. And also looking at how the phenomenon of… A psychologic phenomenon of flow or like being in the zone can also impact creative performance. And that’s been really the topic of my work for the last several years.
0:52:29 SC: So do we know the answer?
[laughter]
0:52:31 DR: I can give you… I can give you kind of the… The abbreviated version, and then we can dig in. So there is two main differences that we have found through these different studies, first, as musicians gain expertise, what we see is when they’re performing, we see pre-frontal activity, deactivating we call this hypofrontality and this idea with pre-frontal activity is associated with conscious controlled thinking, jazz, improvisation happens, and improvisation, I think, is a really interesting topic to study creativity and lens to study creativity through because when we talk about creativity, it’s a really hard thing to define, most people, most people will agree that it has to do with novelty and appropriateness in a given… In a given context, but expertise, and also the type of creativity is really gonna come into play, so there is the creativity of a small child finger painting for the first time, and then there’s the creativity of Einstein. We’re talking about all kinds of…
0:53:54 DR: All kinds of different skills, and so when we are talking about what does it mean to be a creativity, it really depends on that definition, hence in our studies what we did was we looked at creativity with two different lenses, and based on how you categorize creativity, you’ll have slightly different results, so in these studies, the one component was we had jazz musicians come into this study and they would perform a number of jazz solos over a backing track accompaniment they would be randomized and send out to some jazz professors and experienced jazz players, a number of them would rate them on a series of different metrics, one of which was creativity, another was technical proficiency another aesthetic appeal, as you can imagine in a highly technical domain like jazz. There is lots of overlap between those scales, so we ended up, we average them into a composite quality score, right?
0:54:52 SC: Okay.
0:54:52 DR: And So through these studies what we see is that Well experts are performing different, than novices, they are more experienced they have a more robust set of explicit knowledge, and if we go back to those system one system two approach it’s the same idea here is that through rigorous through rigorous training, years and years of explicit practice, we are able to develop these kind of implicit unconscious systems that are able to perform extremely well in these highly cognitive in this high cognitive demanding task, and that’s kind of why jazz improvisation is really interesting, creative task, because in a lot of creative tasks, you have these very distinct phases of the creative process, so if I was… It’s like in terms of talking about whether it’s music or another topic, composition versus improvisation, for example, is… There’s this ideation phase.
0:55:52 SC: Okay.
0:55:53 DR: Generating lots of different ideas. This is related to one aspect of creative thinking, which is divergent thinking, is deep which is of the most tested area of creative thought in terms of standardized assessments is how many different uses can you think of, for a brick or a given object, and that would that’s kind of like when you were trying to come up with a new part for a song on, that’s that ideation phase, I mean composition, but really the ideation phase, and then the next piece is evaluation and ultimately modification in many domains in terms of that cycle of the creative process. That’s more evaluative. Right, and what happens is you’re modifying, you make edits and you’re tweaking, and then you kind of maybe go back to ideation phase and you go through this process, it’s iterative with improvisation you get one shot and it’s all happening.
0:56:42 SC: Theres no time for any of that. Yeah.
0:56:44 DR: Yeah, it is all happening in real time, sometimes as fast as 240 plus beats per minute with cords changing perhaps at every quarter note or every half note, and… So it was amazing in which some of the greatest jazz musicians say things like Miles Davis famous quote, I’ll play and I’ll tell you what it is later seems like it’s amazing, these eminent performers, these anecdotal response to show that they’re actually letting go, they’re releasing that cognitive control to be in more of a state, which I think they’re familiar with the state also in terms of being that experience in terms of getting there and are able to enter that state more easily of relaxing and performing kind of in a bottom-up sensory kind of way.
0:57:37 SM: Yeah, I wouldn’t venture to compare myself to Miles Davis, even though we share the Miles part of the name, but Dave and I had a presentation that we worked really, really, really super hard on, and we kinda did this kind of surprising thing because we’re kind of doing this meta-demonstration of the surprise effect of the music, and when we finally did the presentation, which was like this is in Nashville at the BMI building in front of a bunch of industry executives… Music industry executives. It was about six minutes long, and so there was a musical part of it, and we get to the Break, the musical break after the first verse, that is the course and the second verse starts and I had the feeling… I don’t know if you know Sean, like when you wake up when you’re on vacation somewhere and you’re jet lagged and you just wake up and you don’t know where you are.
0:58:29 SC: All that time.
0:58:29 SM: That… Yeah, so that’s how I felt, but I was on stage in the middle of the performance and I was highly engaged in it it was like I was carrying the performance and I felt like I was waking up, I didn’t remember anything that happened before, and we were three minutes into the performance.
0:58:46 SC: Right and I think that… Help me get it straight here, so there’s this saying that If you’re not an expert, if you have not practiced as much, if you’re just an amateur you’re just learning. Improvisation, maybe creativity more generally, but at least improvisation is more of a system two thing ’cause you’re thinking it, you’re overthinking it or you’re trying to control it very much, and then through training, it becomes a system one thing, it became subconscious and automatic.
0:59:13 SM: Yeah, tell ’em about the John McEnroe thing.
0:59:14 DR: Sure, yeah, I think that’s a pretty good assessment in terms of novices in a domain, I think that we were cueing each other’s stories today. So John McEnroe, yeah, I’m showing my age a little bit, he was a tennis player, really famous in the ’80s, he was very charismatic to say the least, screaming obscenities often on the tennis court, which was very interesting to say the least at that time, but there was a match, I believe it was during Wimbledon, and he was getting beaten by this underdog opponent. He was the… John McEnroe was the top ranked player in the world at that time, and in the first set, he was getting beat and this player, he couldn’t return his forehand, he couldn’t… He just couldn’t hit it back, and so after sets in tennis, players will switch sides of the court that they’re occupying, and as they switched sides, John McEnroe whispered in his opponent’s ear, “What are you doing today with your forehand? There’s something that’s going on that I just can’t hit it back like you really have been working on that.” And then after that set happened and those words were exchanged, John McEnroe beat him in straight sets. And the idea there being that in tennis similar to music, it’s very much… It works on the intersection of really cognition and action.
1:00:38 SM: If you’re an expert. If you’re a expert, yeah, it’s better. That’s the point that Sean was getting at. Yeah, if you’re expert, you can do it implicitly better than you can do it explicitly. But if you’re not, you better stick to maybe trying to figure out what you’re doing.
1:00:51 SC: Right.
1:00:52 DR: Yeah, exactly.
1:00:53 SC: And this goes beyond music, presumably. I mean, is this a general lesson for creativity or mastery that what we aim to do by mastering the subject is taking some of the more system two parts of it, the parts that we really need to think about everything we do and transferring them into system one. And that’s our goal.
1:01:15 SM: Yeah, and that’s actually… Well, that’s actually… One of my colleagues had… One of my… The people who was… The professor, he was on my dissertation committee at Georgetown works in… His… He actually was also with Poggio at MIT, but he works with automaticity and the idea is things going from frontal to further back to parietal lobe, which is on the top of your head, and he sees that happen over time as people practice simple tasks.
1:01:42 SC: Yeah, I think it’s probably obviously true that when we master something, we wanna get the muscle memory or whatever it is, we wanna make it automatic, but it’s interesting to me that we’re now able to actually see it literally in locations in the brain. How advanced is that project trying to see literally something that we would have done with part of our brain lighting up as we get better at it, the same task lights up a different part of our brain.
1:02:09 DR: Yeah, so it’s the technology that we have available to us now really makes that possible and we’ve done our series of studies with both electroencephalography in terms of correlational work. I’ve also had the opportunity to use Transcranial Direct-Current Stimulation, which is actually placing two electrodes in someone’s scalp noninvasively and sending a weak electrical current into the brain, in which you kinda flip the neuroimaging pattern or paradigm on its head and… With EEG or FMRI research and neuroimaging, what you’re looking for is to see what happens in the brain when I either have two groups of participants like novices or experts, or I have two different tasks such as performing a memorized melody or an improvised melody with the stimulation what we look… We say, if we target this brain area and try to modulate the activity in an inhibitory or dis-inhibitory kind of way, do we see a difference in the output of the behavior.
1:03:09 SM: Yeah.
1:03:10 DR: And so we were able to use both of these methods to look at this kind of this hypofrontality and movement to posterior activity, I think it’s important to say the difference is really between the experts and novices. So it’s different groups are variables kind of the number of gigs or how much experience you’d had in jazz. But I wouldn’t just… I wouldn’t be so quick to say that we’re relying only on system one when you’re an expert and system two when you’re a novice because my theory is I haven’t tested this yet, and I would love to do this experiment, is if we look at smaller time scales, specifically when errors happen, and errors and mistakes in lots of the creativity literature associated with breakthroughs of creative performance where a misstep took you on a new trajectory increased uncertainty, and then you kind of you went with it. And then when you go with that, you’re resolving it and that kinda brings it back ultimately to this, the same kind of expectation theory that we talk about on the perception side of things.
1:04:17 SM: Sure. Yeah, it’s like I was talking to somebody today. I was saying that… I was seeing one of your old talks, Dave, in preparing for this and I was thinking like a Formula One race car driver, right? You’re just cruising down the road, you’re just… In the zone, and then all of a sudden somebody checks you or… Like, something happens, you’re like, “Oh my God, wait,” and then you get out of it but you can get the lead because of that, because you make that explicit decision to do something different but I think one of the things that I was thinking of when Dave brought in the whole thing about changing behavior and whether… It’s this kinda distinction between Dave and me in our academic career of kinda trying to use these stimuli.
1:05:01 SM: And use music to try to figure out and discover things about the human brain. I think that’s the overarching dominant paradigm in music cognition. And I think what we’re more interested in, now that we’re running a business but also we’ve… As musicians because we’re curious about the music, is to say, okay, let’s sit back now. We have these wonderful tools that can show us these amazing things that are happening in the brain. But we know what they are. And now that we know what these signatures in these markers are, we know there’s dopamine here. We know there’s certain neural signatures for event-related potentials when there’s error, which is detected, which is like surprise or this tension. So what we’re trying to do now in our next research, and we’re actually working with the National Science Foundation to try to get this done in the near future, is to say, okay, in the things that we’re looking at in Secret Chord Laboratories and the business end of it, to say, “This is what we think the surprise is in the music.”
1:06:04 SM: In this information theory, cave man kind of way. But you know what? That’s not accounting for everything, right? So why is it? And maybe we can get some of that gap closed through music theory. And then we have more of a music theory. But some of that is, we need to actually bring the 13 to 19-year-olds or whoever it is who’s consuming this music, and we actually have to tailor these controlled stimuli. And what we’re actually measuring is in the music. The stimulus that we’re using, it sounds kind of a mad scientist-ey, but the stimulus is these kids brains. We’re looking at what happens in the brains and we’re trying to say, “Okay, what gets that reaction that we know is there?” Because all these… We’re standing on the shoulders of giants, and Robert Satori and all these other… Ani Patel, and us and other people have made these discoveries. We know what it looks like in the brain. How can we get that reaction? You see what I’m saying? So we’re using the brain to learn about music.
1:07:07 SM: And it seems like that helps us. But it also seems like there’s not enough of that in academia even, and we’re continuing to publish these results. And so because everybody’s looking at juicing this kind of stale controlled music to try to see what happens in the brain. It’s not necessarily still, but because of how science works, you have to only change one thing.
1:07:28 SC: Yeah. And it brings up a related issue that I wanted to get into because we talked about improvisation and the test that you did, David involved individual guitarists listening to a pre-recorded background track. But another aspect of improvisation is that you are in a group, right? You can be on the stage with a bunch of people, improvising separately. And I’ll confess as we’re recording this, I’m just a few days after doing a webcast with Wynton Marsalis.
1:07:53 DR: Wow.
1:07:54 SC: Wynton was one of my first guests here on Mindscape. So he returned the favor, he had me on his webcast. And he’s just very interested in the science or whatever we can learn about how people are interacting there on stage with their creativity. And certainly making mistakes was a big part of it. Recovering, being surprised. Sometimes you’re in-sync, sometimes you’re exactly out of sync. People are doing opposite things and sometimes that’s okay. How far have we gone in studying how the different brains of different musicians are interacting with each other up there in real time.
1:08:28 DR: Well, first, feel free to give Wynton a call for me. I’d love to have him and his group in neural imaging study. It’s interesting that you say that. So coming from… Growing up, I was a big fan of the band, Fish. Rock improvisation, that’s where… It was my access point to Improv before I got into fusion and jazz and other styles of improvisation. And I think you really hit the nail on the head, is that what really drives that live performance with a group is when the synchrony happens. The whole building feels it, when the four or five members are in-sync together. And there’s some preliminary work. That’s really where I kind of found that you could even record guitarists with EEG signals and the signal wouldn’t be disordinate enough. There’s a group in Germany who looked at dual guitarists and improvisation, and having them improvise together. Now this is… As you add more musicians, four, five, six, you add more complexity. More things are changing. So they went from one to two and looked at duets.
1:09:41 SM: Not to mention looking at the perceivers, right? You would also wanna see if there’s any synchrony there with the audience and that would be really special.
1:09:51 DR: Right. As far as I know, there’s two places in the world that have the types of facilities where they have an auditorium with neurological and physiological metrics for an audience and musicians on stage of McMaster University.
1:10:05 SM: Toronto?
1:10:05 DR: Yeah.
1:10:05 SM: There’s one in Toronto.
1:10:06 DR: In Canada and the Max Planck Institute in Germany, which actually before I joined Secret Chord Laboratories, I was gonna head over to Germany as a postdoc. And then when we realize that this kind of research… ‘Cause we could steer the ship on what exactly, what kind of music we choose and artists and how we design our studies, that’s definitely on the forefront of… Our research goal is further exploring the interactions both between band members and also the audience as well.
1:10:39 SM: You said you’ve gotten so far as to get to Phish’s management for that.
[laughter]
1:10:43 DR: Yeah, so it’s funny. Scott and I were attending the Northeast Music Cognition Conference, I believe it was back in 2017. I think it was the week between my wedding and my honeymoon. And I gave a talk there, and afterwards a big Phish fan up in Massachusetts came up to me. He’s a medical doctor who uses Ron Hirschberg, who looks at the clinical aspects of music and how it can affect the rehabilitation of military vets.
1:11:10 DR: And he had been… He was a huge Phish fan, and he had been recording… He would record it with a group of 100 people some background vocals and told me this whole story from 1992 in an old church. He came up to me, he’s like, “All I could think about… ” Now, I didn’t mention Phish or anything during my talk, it was focusing on jazz, as we’ve been talking about. But he said, “All I could think about during your talk was doing a study with the four guys in Phish.” And I was like “oh, my God,” and then we just started talking and going, “I know their management.” So then we wrote a letter to their management. We included all of our CVs and our cred and our post publications, and they were like, “We’ll forward it to the band,” and then you fast-forward a couple months later, there was actually an academic conference called The Phish Studies Conference in Oregon, and I got to go out there and…
1:11:53 SM: You had coffee with Mike Gordon, right?
1:11:54 SC: That’s good.
1:11:57 DR: And proposed this study of getting basically all four guys hooked up to EEG devices. The drumming data probably wouldn’t be so good, but actually looking for those peak moments, which is really what the work focuses on, is those peak creative movements. There’s quite a bit of research looking at improvisation just generally as an activity versus other forms of musical performance like composition or playing a memorized melody, but I think that’s a consistent focus, and that’s where flow comes into play, is there’s distinct peak moments that if you asked everyone involved who was performing when were those, I believe there would be high agreement, and then to look at the underlying brain data and musical data is definitely a goal of mine.
1:12:41 DR: Another fascinating aspect of the results from this study is that we found critical hemispheric differences between the left and right hemisphere. In the mid-90s, Elkhonon Goldberg developed and tested the neuro-cognitive theory which showed evidence for the left hemisphere being more engaged for routinized and habitual tasks. The right hemisphere is recruited in more novel and unfamiliar situations. So in our study, expertise, or jazz experience, was significantly predictive of the quality of an improvisation. Thus, when we first analyzed the brain activity associated with high and low-quality improvisations, the high-quality ones showed us both posterior shifts in brain activity and left hemispheric activity. So if we were to define creativity by this product definition, which is what we’re doing when we’re asking experts in a domain to rate the improvisations, the left hemisphere would be critical because the experts are likely using well-learned routines and a jazz vocabulary and phrases they’ve learned over many years to construct a solo on the spot to new chord changes.
1:13:49 DR: However, once we’ve statistically controlled for expertise, then the brain activity shifted from the left to the right hemisphere, indicating that that neural activity in the right hemisphere underlies the more creative improvisations once we remove the effects of jazz experience. So these results represent more of a process definition of creativity, where novelty and flexibility are central. So generally, if creativity is defined in terms of the quality of a product, like a song or an invention or a poem or painting, then the left hemisphere plays a key role. But if creativity is understood as a person’s ability to deal with novel and unfamiliar situations, as is the case for novice improvisers, then the right hemisphere plays a key role.
1:14:39 SC: The other thing… On the flip side of this, you’re both musicians in some sense, right? I don’t know… I’m not familiar with your work, per se, but I believe that this is one of the things you do. How much does this academic, scientific study of what’s going on in the brain, etcetera, change how you are as musicians? Does it affect how you play, how you enjoy it, how you think about your success?
1:15:02 SM: So it’s interesting, so, when I came up with this, I was kind of a disgruntled… Not disgruntled, but I was… Anyhow, I was having a great time working for a Harvard neurologist in Boston, and I was also work… I was kind of not focused, I’ll put it that way. And I was working on this huge album project, and it was my first album, and I was trying to do everything independently and perfectly and professionally, and I had the Boston String Quartet, and I rented out like WGBH for the Frazier Studios for their big Steinway Model D make here, and everything was perfect and purist and that sort of thing. And that’s really when I came up with the idea. One of our co-founders, Brian Owens, who’s back here in Virginia, I kept sending him, “What do you think of this bridge, what do you think of this bridge?” He was like, “I don’t know, they all sound the same.”
1:15:55 SM: And so I was like… I was doing fMRI, and so I was looking at fMRI and language and auditory processing, and I was like,” I could put this all together” and that’s how I came up with it. The short story for me as a musician is really working on this 60, 80 hours a week for the next 10 years after that, that was 2011, right? Has not only occupied my time, but also discouraged me from writing new music because I keep saying, I’m very close to knowing everything there is to know to writing the perfect song, so I’m just gonna wait and I’ll put my ideas together then. And it’s giving me an excuse to procrastinate from the creative side of my musicianship.
1:16:40 SC: Yeah. There is no such thing as the perfect song, sadly. It’s the same thing with writing or anything like that.
1:16:46 SM: Of course, yeah, but it’s just the idea that, “I’m gonna have a lot more data to base off… Why would I write something now when I could write it with the benefit of having all these insights from all the greatest songs ever written?” So that’s in my mind justifying my procrastination, but could I still jam, since then?
1:17:14 SC: Sure. And, but finally, we’ve both mentioned… You both mentioned several times that the concept of flow, being in the zone, feeling it. To what extent can we extend this beyond not just music, but even art all the way into our lives, right? Are we learning something here about how to live well, not just play music well, is there something to be said for becoming such an expert at life, that it becomes second nature to us and we’re in flow and in the zone?
1:17:45 DR: Sure. That’s the philosophy that I’ve taken. I think flow… I mean flow is related to positive psychology, it’s a theory, that’s where… That’s really where it comes from, as a mental state where the individuals are immersed in a task and there’s this intrinsic motivation that really, you choose to do it because just the experience and having that energized focus, it’s a joyful process and you’re totally absorbed, cognitively, and I think this definitely expands to… I talk to my friend who codes video games to people who enjoy hiking, reading is a task that’s in high flow, and for me, I keep my sanity by, when my yoga routine or music practice, practicing with with my band falls out or writing, and I lose those pieces for because work comes into play too much and I get too caught up in other things, I notice the effects. I come higher in anxiety, maybe a little more depressed, and when those pieces fall out of line, and I try really hard to…
1:18:54 DR: I think that’s like a, there’s a discipline piece also of saying, you don’t always see the output of writing your song and then having it be commercially successful with or without the formula, but that there’s another component of just engaging in and developing a craft regularly, which I think makes these flow experiences more common and easier to enter into.
1:19:22 SM: Yeah, and Dave brings up the… It seems kind of paradoxical with flow and routine, right? And so, you think about people like Steve Jobs or Mark Zuckerberg or where you do things routine… A lot of things that we do in the company like are, we just like that I’ll do, that people will say, “Why are you so obsessive compulsive about certain things?” And to me, it’s like, you know what? I don’t wanna have to worry about what I pick out, like Steve Jobs in the case of wearing the turtle neck, right? I don’t have to worry about fun… There’s no right answer. To me, it’s like the type of thing that there’s no right answer, so why occupy your mind if you’re a big, brilliant person? And then that way if you don’t have to worry about certain things, if they’re all routine, if you are sticking to a routine, the things that you are focused on, then you can riff on. And that’s the kind of thing that when we’re talking, obviously, it was a little bit tongue in cheek about waiting to have the pattern to write the song, but it’s like jazz, right?
1:20:21 SM: So when you have a lead sheet in jazz, you know that the cord changes for September Song or whatever, the cord changes are gonna be fire, they’re gonna be great or if you do it like a 2-5-1 or whatever it is, you know what the cord changes are, and if you have certain constraints, then you’re able to have more of a flow state in everything else you do.
1:20:47 SC: Yeah, I gotta, but now I don’t like it anymore because I think that this is a… You’re sacrificing something for something else, sure, but I don’t wanna be living in a world where in order to be the best software designer, I wear the same clothes every day. I think that, you know this is…
1:21:05 SM: Right, but I mean you have people who have secretaries who take care of their laundry for them, or take care of their phone calls, you know what I’m saying? So they can just… Like the Rolling Stones, I read this book one time, it was like, why are the Rolling Stones so great? Because they just get there, they get out of the bus, they go on stage and they play.
1:21:21 SC: Right.
1:21:22 SM: They don’t tune the guitars, they don’t set up the gig, they don’t do all these things, and that’s another thing you know, when you get into the whole tension about the artists are having to do everything now, because there’s no artist development in the music business anymore, but there’s a certain thing to be said for only doing what you’re a peak performer at, you know afterwards…
1:21:44 SC: There is. No, there is, and I’m very… I’m being legitimately and I’m not playing the devil’s advocate now, I’m uncertain what to think about this, ’cause this is a very provocative fact because…
1:21:54 SM: Sure.
1:21:55 SC: The flip side of that is, if you’re the one who tunes the guitars, you know your guitar a little bit better, right?
1:22:01 SM: Sure.
1:22:03 SC: People are always surprised when I invite them on the podcast and they’re not familiar with it, and they’re like… They mention my producer or my editor, and I’m like, “Yeah, that’s all me. There’s no person… “
1:22:14 SM: Sure, sure.
1:22:15 SC: Who is the audio engineer here, this is all me. And…
1:22:19 SM: And yeah…
1:22:19 SC: There’s definitely pluses and minuses to that, right?
1:22:21 SM: Right. Oh, yeah.
1:22:21 SC: The audio quality is not as good, but at least I understand it, what is going on there.
1:22:25 SM: Well, I certainly don’t have any of those people, so yeah…
1:22:27 SC: Yeah, exactly.
1:22:27 SM: I don’t have a secretary. Yeah, I get it.
1:22:29 SC: Yeah, well, I think…
1:22:29 SM: Dave, did you wanna say…
1:22:30 DR: I think when we talk about flow in this very general sense, we’re not really talking about how it’s operationalized in terms of the psychology literature. And the work on flow is… There’s very little neuro-scientific work on it and it’s not well studied, and I think part of that is because of the complexity, there’s nine kind of different components. And so people look at… They break those into two subsets of components. We can talk about there’s conditions that allow flow to happen and in the external, so those two main features there are of…
1:23:09 SM: Is this Csikszentmihalyi?
1:23:10 DR: Csikszentmihalyi, his book, Flow 1990s, kind of seminal publication there, but there’s been a number of papers since then and so there’s this balance between challenge and skill as one of the conditions, and then clear goals and feedback. So those are the conditions, that’s how it’s set up. So some researchers, they choose one of these aspects such as the balance between challenge and skill. I know I’ve seen some work where you set up… They have had people come in and do math problems, and depending on your math ability, they would create an algorithm that would present math problems to you, and how you, if you get a problems wrong or correct they would advance you or move you back on the level, and have you hooked up to some type of electroencephalography and look for neural correlates of this facet of flow. When we talk about this in terms of music, I think what we’re miss… If we’re defining flow only in terms of balance between challenge and skill, which is this condition of flow, we’re missing out on the key phenomenological aspect of flow, the subjective state that people report and experience and on that side, we talk about, those dimensions include intense concentration and immersion, the merging of action and awareness, a sense of control, the distorted sense of time that Scott mentioned earlier, and critically, I think the experience of intrinsic reward.
1:24:31 DR: So for studies that would only, that would encompass flow as only a balance between challenge and skills such as the math task, I would ask you, “Well, how much do those people love math?” Right? So it’s a…
1:24:45 SM: Right. I think yeah, like a lot of this comes down to who would come to where you are in a given day and say, “You’re lucky to be doing what you’re doing for a living,” or have that hobby or whatever it is, because I think that’s what people think of when they think of somebody who’s lucky to have a certain job or if they’re doing something that they’re good at, but is a little bit challenging, and not everybody finds that, and I think as an educator, which I know Dave has spent, in his other life he’s spent a lot of time in STEAM education and that sort thing, I think we have a long way to go in helping students, children and helping them find the place where in the world where they’re gonna be beneficial to society, be able to feed themselves and also somehow reach this kind of zone, you know? At the most possible.
1:25:35 SC: So what I conclude from this is that life and success and creativity are complicated and difficult, but that it’s possible to make progress and learn about them, and I love the idea of using music as sort of a paradigm for studying all these things in the brain, so David Rosen and Scott Miles, thanks so much for being on the Mindscape Podcast.
1:25:54 DR: Yeah, thanks for having me Sean.
1:25:54 SM: Thank you Sean.[/accordion-item][/accordion]
Pingback: Sean Carroll's Mindscape Podcast: David Rosen and Scott Miles on the Neuroscience of Music and Creativity | 3 Quarks Daily
Terrific podcast content, Sean. But the McMaster University facility “where they have an auditorium with neurological and physiological metrics for an audience and musicians on stage” is in Hamilton, Ontario rather than Toronto.