224 | Edward Tufte on Design, Data, and Truth

0:00:00.2 Sean Carroll: Hello everyone, welcome to The Mindscape Podcast. I'm your host, Sean Carroll. The world we live in today, and I don't think anyone can argue is suffused with numbers, with data, with information, data and information come at us from all directions, whether we're reading a newspaper or magazine, looking at something online, watching TV, scientific papers, and the thing about numbers and data is that they have this aura of objectivity. Like there's a number you can't argue with it, it's giving you some information that could be false, but as long as the information is reliable, it's telling you something objective about the world. But the reality is that the way that we present that information visually, whether it's in literal series of numbers or a chart or a graph or whatever matters. It affects how we process the information. What seems important to us? What is it that we notice? What is it that we care about? So sometimes you want to do your best, I hope that usually you want to do your best at conveying information clearly and vividly and concisely. Sometimes maybe you wanna feeble a little bit right and hide the parts of the information that you're not that proud of. 0:01:09.5 SC: The one way or the other, it matters how it is presented, so the art and Science of presenting information is very important in the modern world, and today's guest is the guy when it comes to the display of quantitative information. Edward Tufte is the author of the classic book, namely The Visual Display of Quantitative Information. The really pioneering text that help people understand the importance of graphs and charts and how they are presented in the way to do it well, and since then, he has continued to try to educate as many people as possible about thinking clearly and presenting those thoughts clearly in a visual form. He has a new book out called Seeing with Fresh Eyes: Meaning, Space, Data, and Truth. 0:02:00.6 SC: And once again, it's an exploration not just of how to present information, but the meaningfulness of that information. One of the big things that Edward pushes is that the origins of quantitative information as a way of talking about things you can trace back to, let's say, Galileo for example, and it's not just a new way of presenting information, but a new way of thinking, a new way of arguing for a conclusion based on evidence, rather than just words giving you an argument, a shift from rationalism to empiricism, if you like. And in the new book, he talks a lot about truth, there's a lot of physics diagrams in there, as well as a lot of works of art, which give you a different kind of truth. So it's a real pleasure to talk to someone who is truly a major transformative figure in their field, and just listen to the wisdom that he has to offer. So let's go. [music] 0:03:04.1 SC: Edward Tufte, welcome to The Mindscape Podcast. 0:03:12.2 Edward Tufte: Welcome. Good. 0:03:13.6 SC: I do wanna start with the sort of obvious caveat/apology, which is that you're the world's expert in data visualization and we're doing an audio-only podcast, so I do understand that it'll be even nicer if you could point to things, but we'll have people check out your books and so forth, you have a new book out. Why don't we give you that chance to mention that. 0:03:35.1 ET: Okay, it's a volume 5 in this series that started in 1983 with The Visual Display of Quantitative Information, and this is the fifth book. The title is quite immodest, Seeing with Fresh Eyes: Meaning, Space, Data, Truth. And I can't think of a grander title, but I kinda set it as a goal as I was writing, I had to talk about all this stuff with this the title and that's exactly what it's about. It's redesigning everything, sentences, paragraphs, Feynman diagrams, everything, sculpture, and that requires the seeing with fresh eyes. I open a book with a little bit of free verse, which describes it and also describes me, if it's all right, I could... 0:04:44.8 SC: Please, we'd love it. 0:04:46.1 ET: This is a free verse, it's called The Thinking Eye. To see the ordinary so intensely, that the ordinary becomes extraordinary. Becoming so focused, so specific about something, that it become something other than what it ordinarily is. Always on, thinking eyes see intensely, actively, skeptically, scan globally, focus locally, see a varying scales of space and time, approximating ways through multiplicity, detecting how things happen, move, act, interact. Seeing with fresh eyes, vacation eyes... I love that. I love the vacation eyes. We have a lot... The first five days at a place, it just seems so in like a wonderful fresh eyes. Seeing with fresh eyes, vacation eyes, unhindered by self-confirming words, models, expectations. Not seeing something different is not seeing anything at all. Grace Hopper saying, The most dangerous phrase in the language is, we've always done it this way. Staying in optical experiences, forgetting the name of what one sees, this is a very important idea about seeing with your eyes, not seeing with words. Once you have the words, it's impossible to see around them. It's very important in sculpture, for example. 0:06:24.8 ET: If you walk out and see a big thing and people say, Oh, it looks like Stonehenge, or it looks like this, or it looks like it was... Somebody bought it for a million euros in Zurich, the other last week at an auction. You can't see it any other way. And so that's, the scene has gotta be free of words because the words will dominate what you see. So stay in an optical experience forgetting the name of what one sees, lusting playful eyes. Shut up and look, if you see something, say nothing. De-familiarize, de-contextualize, re-contextualize, reform, remodel. Thinking eyes are of this world. Empirical, specific, practical, self-aware, asking, dis-believing, challenging, making the familiar unfamiliar. 0:07:18.4 ET: How do you, I, they really know that? I's your little stack list in the middle of how do in the satin stand vertical? You, I, they, really know that? How could you, I, they possibly ever know that? That is how could you design a research. Could you even design a piece of research that would enable you to own it? Thinking eyes reason intensely about what they see, reason about verbs, links, mechanisms, connections, dynamics. Reason of not what things do, not what things are named. Reason across multiple time horizons, now, then forever. Name, rename, remodel. Thinking eyes compare, model, choose, doubt, decide, compare again. Thinking eyes act, make something of seeing and reasoning, discover, produce, construct, write a report, make an art work, teach a class, have an insight, understand, explain, show, get on with it. 0:08:31.5 ET: To produce, construct, model, remodel, to act is essential. It is a difference between spectator and player, between consumer and producer, between art chat, versus art work, anecdote versus evidence, process versus outcome, retrospective versus prospective. Presentations pitching versus demonstrations comparing. Craig Venter saying, "Good ideas are a dime of dozen for a smart person. What distinguishes good from great is how an idea is executed. How it becomes reality, thinking ideas, thinking eyes identify, know, celebrate excellence forever universal knowledge. Gathering consequences, staying in place beyond memories and precision, seeing, learning, doing doubting are the meaning of intelligent life. 0:09:33.8 SC: I like that, I feel like I should applaud a little bit for the performance. Thank you. But I'll say that I love the word truth appearing in your subtitle because that's what hit me over and over again, sort of reading through all your books over the decades and thinking about what you're doing. There's this sort of down to-earth operational side of making a beautiful chart. And then there's the much, more profound question of discovering truth, presenting it and conveying it, and that's really the motivating factor. Yeah? 0:10:06.3 ET: It's what it's all about. It's in any good report or presentation, it's about the content and the credibility. It's not about whether you should do a certain motif or use a certain method, it's about the content and the truth of it, the credibility. And that's what's going on in information exchange. And this kind of stuff about the coding for visualization and all that, I regard that as plumbing. Because really, it's plumbing basically. And the reason we're having the reading that research report or that we're having this meeting is to reason about content and assess the credibility of the material. That's what that's the fundamental thing it's all about. And all the kinds of things I did early on, there was this concern with getting it right, getting it true. 0:11:14.8 ET: But it's really as I've become more general and applying to more and more things, truth comes right along with it. And the plumbing may differ, but its content and then are they saying something true. I have a long chapter on medical research in the new book, and it's widely agreed by editors of the General Journal of Medicine, by editors of the Lancet and by the famous skeptic John Ioannidis at Stanford that most published Medical research findings are false. And the debate is on over whether that statement is true or false, the debate over is... The Debate is over the word most. [laughter] And so in the field, it appears to be at least 50% in published medical research. And this is why people have been editors of journals, and it varies by fields. In some fields I did the Ig Nobel Prize Awards this year. 0:12:30.7 SC: Love those. Yeah. 0:12:31.9 SC: Yeah and I did 24 jargon words about cognitive psychology, and then you have seven short words at the [0:12:40.0] ____ I went through the Jargon of parameters, blah, blah, blah and so on. And then the punchline is that the first day in the class in cognitive psychology, the professor says, "Half of the findings in our textbook are false, we just don't know which half. [laughter] So it's part of the failure to replicate. The replication process. Science has it easy. Rocket science compared to social science or even medical science is easy because you've got a guarantee of the truth. There is truth in the laws of nature. And that makes it easy. You know there is truth. You don't even know whether truth exists about a lot of personal... People things, or medical things. So, rocket science is easy. Compare... 0:13:45.5 SC: I say it all the time. I tell people that physics is the easiest science. That's why it's so intimidating 'cause it's so easy. We've learned so much about the physical world. 0:13:54.6 ET: It's extraordinary, and also the only thing that's universal. It's through everywhere. That's out of this world. [laughter] 0:14:04.8 SC: So let's make sure that the people... That the tiny percentage of listeners who aren't already familiar with your work sort of get the thrust of it right from the start. You work at the intersection of data and design, and design is a tricky thing. Do you have any training in design, or is this something that you just built up along the way, or did you stumble into it, or was that a goal all along in your career? 0:14:32.3 ET: My parents. My mother was a professor of English and did some scholarly work on the 17th century, but she also wrote a book which I then published. I for myself published, or I published my mother's book called Syntax as Style. It's called the Artful Sentences: Syntax as Style. And so that was kind of the word part in my home. My father was a civil engineer, which is very Applied Science. 0:15:07.6 SC: There you go. 0:15:08.3 ET: And it's outdoors, and that helped me a lot in my sculpture, of course. And they both could really see well. I don't know where that came from, but they could really notice things and see well. And I was taught to see well, both in reading things and reading poetry, and looking at pictures and talking to my mother, but also being outdoors with my father. So we'd take a vacation, we drived for Hoover Dam in the middle of the Colorado River, and study the dam, because my father was a civil engineer. Or every time it rained, we would go out driving to see the storm drainage scurry off the wrong way. So intense seeing was just part of it. 0:15:56.6 SC: Interesting. 0:15:58.4 ET: And I married the famous graphic designer, Inge Druckrey, who was a professor at Yale and RISD, and the University of Arts in Philadelphia. And I learned a lot from her, and I found design very easy. And as soon as I did my book on visual display, I was teaching in the Yale Graduate School in Design. So college was good enough. [laughter] 0:16:33.6 SC: Didn't need to have a degree. 0:16:35.6 ET: Because I had done the design for the books by working with a book designer, and then the rest of the books I've designed entirely myself. And frankly, I found design very easy. I think it was because of my verbal skills. I think many designers are like violinists, or there's a kind of innate quality of seeing and working with your hands and making things. And it's not so much... It's a kind of almost a physical performance, not so much an intellectual performance. So there's a real difference there. It's a seeing performance, not a word performance. That's a better way to put it. And I'm a B plus in a lot of different fields. I'm an A plus in visualization, but a lot of the sub-fields like writing, design, statistical work and all this, I could probably teach an introductory course at the school. And I can use right now these tools that I can use all the time, and I can do everything myself, within myself. So the books, I design myself, I publish them myself. I love the craft and the former craft of doing books with the computer screen, it's not like that anymore. And so it just somehow came in this odd package of genetics and school, and never specializing. That's actually that was the key thing. I got a bachelor's and master's degree in Statistics from Stanford, PhD in Political Science at Yale. I was a good political scientist, I was a full professor by age 31 at Princeton in Political Science, but it was quantitative. 0:18:48.9 SC: Oh yeah. [chuckle] 0:18:49.0 ET: I was looking to go over to graphics more and more. I had started a politics department while I'm supposed to be doing this, and what's [0:18:55.2] ____ this graphic stuff. And I think I always wanted to be a professor from about age 12 on, but I have yet to discover, a professor of what. 0:19:06.8 SC: Right. 0:19:10.0 ET: I taught in the Yale Law School. I taught in design. I had tenure in public affairs at Princeton. I had appointments in statistics, and maybe it's 'cause I have a short attention span, but I loved going into a discipline and looting it. Not getting a degree in it, but rather looting what was useful for me. And I get in a completely different posture when I talk to people in other fields who were... And I've always been attracted by excellence, by hanging around excellent professors when I was a student, regardless of what they were excellent in. And seeing how they think and just being with them most happy. And it's the only time I really shut up. [laughter] It's when in this environment where there's all this stuff to be learned. And I just say, interesting and Oh, that's interesting. And maybe say, guide them slightly ever. Maybe say, why is that once or twice every few minutes. But I just love that kind of discussion where I have no responsibility except to listen and guide it just enough to do it. And, it's, I think when I'm happiest, I'm a stranger in all of these contacts, but I'm sort of taken by it. And because I did the data stuff, I could play in just about everybody's backyard. 0:20:51.0 SC: So let me, if it's okay, get down to some nitty gritty. 'Cause I do wanna make sure that every listener to the podcast comes away making slightly better charts [laughter] than they did coming in. I mean, we live in a world where there's graphics and data visualization all over the place. This is probably an unfair question, but what do you think is the biggest flaw or the most common mistake in how people make charts and present their data these days? 0:21:19.9 ET: Multiplicity. 0:21:20.9 SC: Okay. 0:21:22.7 ET: They try five times, five different graphics, or maybe 10. They have a programmer it does is something special and they cherry-pick the results. And my first piece of advice to any researcher is use utterly conventional graphics that are in the very best graphics in your field. And you specify those graphics in advance. You can't search through, you can't cherry-pick. Things are so bad that in medical research in... In clinical trials, RCTs, Randomized Control Trials, they have to specify their graphics before they see the data. 0:22:12.1 SC: Wow. 0:22:12.7 ET: Because everybody was cheating [laughter] Cherry-picking because most everybody is Dr. Confirmation bias. 0:22:21.8 SC: Sure. 0:22:22.8 ET: All researchers are have a little bit of confirmation bias in them. Of course they do. The real giveaway is that in the... There's a study is cheated is that they'll have an unusual custom graphic. 0:22:36.2 SC: Interesting. 0:22:37.5 ET: Not a conventional graphic. And they're so proud of it that the title of their paper is Ending Metastasizing in Cancer by the use of artificial deep intelligence, six point CP2. So they're pitching their finding and their contribution that they've also made to graphical things. [laughter], That is the sign of a fraud and somebody who's cheating if they have a, any kind of decent substance finding, they don't, doesn't matter the plumbing, they're trying to say, well, I brought this thing into the field. I used to referee all kinds of papers. People would say graphics for sociologists, graphics for psychology, and they would think what bullshit. [laughter] It's the principles of graphics are the, in, don't come from the field, they come from the problem, the data problem to be done. 0:23:49.2 SC: Right. 0:23:49.9 ET: And so you do whatever it takes in any field. What graphics did you use on psychology? Whatever it takes is the answer [laughter] No. Not that we have some kind of special things. So the giveaway is an unusual charting method, often custom. 0:24:11.2 SC: So are you thinking about something like, you know, pie charts versus histograms versus line charts? Is that the difference of choices that we're thinking of here? 0:24:19.1 ET: No. We're thinking of scientists more scientific, serious things. 0:24:23.9 SC: Okay. But what are the kinds of choices that you're saying people make to sort of cheat a little bit? 0:24:30.0 ET: Oh, they take... They put things on, they put a Y on algorithmic scale. 0:24:37.3 SC: Ah, okay. Yeah. 0:24:39.1 ET: They, that's the... And there're good reasons usually for it, but not always. If there's a doubt they should show both, that's how you get around it, that your finding survive both transformations, it's the conventionality is also good because the person has come there to learn about, that the cancer metastasizing can be stopped. They haven't come to learn about your graphic. 0:25:13.2 SC: Right. 0:25:15.0 ET: And you wanna minimize, you want them to... Your readers to see the data instantly. Not decode a graphic, not have little color codes like R does endlessly, Python is better on that. And so use conventional things just as you used, there's all kinds of conventions about the language that you use in published papers. They're conventional things. Let's get to them. Get it all, get all that out of the way. And that now allows cross research comparisons. And people who are shopping around for something new are thinking they're making a contribution. Look, Don Knuth did it 25 years ago. Okay? Okay. [laughter] 0:26:03.6 ET: That's true of everything, by the way. [laughter] 0:26:04.8 SC: Yeah. 0:26:06.6 ET: He did my sparklines, which I was so proud of. He did something called skylines, little things post silhouette skylines in a online, an inline graphic. And what really was 25 years before my sparkline. But it's true. Everybody will, once they hear that, they'll say, yeah, that's right. 0:26:26.3 SC: I mean, maybe tell people what a sparkline is for those who don't, haven't heard of it. 0:26:30.8 ET: It's an inline graphic. It's a word inline and it has a resolution of typography. Which is an intense resolution, the letters. So, it's very high data and it's embedded in the text itself and shows lots, lots of it. And it's perfectly readable. And it's the highest res... If you can operate at a graphic, at the resolution of typography, you're in the big leagues. And so that's the metaphor instead of a word, it's a graphic. It's words-like sized. It's built right in. And you have a bunch of these and now you have what's called a table of lines like that with consisting of words and numbers, and also maybe little tiny images. I first got the idea from Galileo. 0:27:24.1 SC: Pretty good. 0:27:25.3 ET: Where I've gotten a lot of ideas. [laughter] 0:27:27.0 ET: Or when he discovered the rings of Saturn and he says, Saturn looks like this. And there's this little charming picture, little drawing, line drawing that the printer had to cut through out of the lead. Make a special Saturn letter. [laughter] 0:27:43.8 ET: And you can see how it was kind of done roughly. And it says, Galileo says Saturn looks like this. And two lines later he says on a cloudy night, it looks like that. So comparison of a clear and a cloudy, and you could, kind of there was a difference. And it's right there in front of you and it's perfect. You don't need it. He that's working at, you know, he could barely see them himself. And I'd, that's showing up and kind of sit quietly. And I think all five of my books, I have Galileo Saturn. 0:28:18.5 SC: Yeah. 0:28:18.8 ET: Because it is so wonderful. He is such a visual person and just seeing, and he is the person who saw more than seeing, he said, we now have the evidence of the eye, not the evidence of the church sitting around in armchairs parsing Aristotle and parsing the Bible about astronomy. And Galileo knows exactly what's happening. He says, we now have the evidence of the eye. We have visible certainty and ding ding, they're all about truth. 0:28:55.9 SC: Well and empiricism. Right. I mean, I noticed that connection also. 0:29:00.3 ET: Yes, it's, you're seeing. The evidence of the eye, not the evidence of words and of churches and all the rest. 0:29:07.3 SC: Do you think... 0:29:09.1 ET: And Eric wrote about it? I've almost quoted, well that what he said. And that's been the spirit of, since book two, Galileo showed up in the second book and Galileo Saturn gets in every book. [laughter] 0:29:23.7 ET: It's about empirical evidence of the, of his eyes. And he is just a piece, he's just beyond everybody. He's a mutation of a mutation. He's so incredible. 0:29:39.6 SC: He was pretty good. I do admit that. And... [laughter] 0:29:44.4 SC: There's a flip side, right? There are graphic choices that as you alluded to kind of make things worse, kind of hide what you care about. And one of the other things again, going back to the truth issue that I noticed in reading the books is, this idea that good graphic design just flows naturally from clear thinking about what are the causal relationships? What are the variables that matter? If you really think super duper clearly, maybe your graphics will pop out the right way. Is that an exaggeration or do you think that's more or less right? 0:30:18.5 ET: This is from the green book, the, with the dog on the cover. [laughter] 0:30:24.4 ET: Beautiful Evidence. And I wrote exactly about that. The purpose of an evidence presentation is to assist thinking. That's the key thing. Thus presentation should be constructed to assist with the fundamental intellectual tasks of reasoning about evidence, describing the data, making multi-variate comparisons, understanding causality, integrating and diversity of evidence, and documenting the analysis. Those are the principles of seeing evidence. But they're also right now, just turned into principles of analytical design. The point of a display is to assist the viewers, that reasoning about it. I can tell you what reasoning about data is. I turn them into design principles. So your design principles are, not show the data, show multi-variateness, show mechanism causality, show an integration of different kinds of evidence and provide documentation. This is a very powerful idea that people don't think about, don't realize. They talk about why we should use this method, whether we should use bullet things and stuff. No, you go deep down. You're trying to support the thinking of the viewer in understanding the data. And I can tell you what data thinking is and turn them into principles. 0:31:51.4 ET: This is a... It's a big idea because it's making now a more of a science of this. These are principles of scientific inquiry as well too. When you're showing that, and so the principles... This is the grand principle of analytical design. The principles of analytical design are designed from the principles of analytical thinking. And people don't act like that. [laughter] 0:32:14.0 ET: They say, "Oh, hey, we can do donut graphics, we can do donut graphics now." Well, how does that help the thinking of the viewer. 0:32:21.3 SC: Right? 0:32:22.6 ET: Does it testify to causality? Does it make comparisons? Does it deal, tell the truth? And so it makes it a completely different thing rather than, "Oh, the new R donut graphic's in it," it gets to be the field now, unfortunately, of data visualization is becoming more about itself than about helping people understand data. It's like the economics department is about economics, not about the economy. That these things become about themselves and this is what's happening in the packages. And they're comparing each other and they make... They often, you know, get the lowest common denominator. So everybody has a donut. And I think that's a sin that... Hear it and my advice for everybody, the 200th publication of logistic regression is that every major graphic should have a package insert with it, like drugs to pharmaceutical drugs, and that they have warnings. [laughter] 0:33:43.8 ET: For example, never make a causal inference from logistic progression, progressive multivariate. Okay. And that, it's so serious that there's a black box around it from the FDA. [laughter] 0:34:00.9 ET: I wrote that up for my new book. And that should come. Box plots, for example, are enormous censoring of data. It's called binning. And it's two-dimensional binning when you have a row of box plots. And they... The drug companies hide stuff all the time with box plots to show the data dots. 0:34:21.6 SC: To show the data, Yeah. 0:34:24.1 ET: Didn't say, because we have high resolution screen cell, we can see the data. We don't have to use these summary things. And so the box plots things don't, don't trust anybody who's using them. 'Cause they have cherry picked those like crazy. They choose the bins, they choose the fact that there's a box plot and they can find fake breaks, turns, ups, and downs, which require a whole lot more data to model that. To have to go up and to start having a polynomial thing instead of just a straight line. And it's a, and so you get these cheap things. There's a plateau if it happens at this point when they're taking the drug and we have to add a whole lot more there. 0:35:09.9 ET: The thing I've most discovered from a viewer's point of view is most strongly in the new book where I spent 40 pages on medical research from a statistical side. And we've gotta be inherently deeply skeptical of human research because of the replication crisis, because of all the false papers and medicine of a lot of cheating with the west... With the blot... I don't know if you know about this, the reading of blot, western blot tests and they Photoshop the same blots in several times and in some molecular biology journals, it's the Photoshop shows up in like 4% of the published papers. 0:36:02.2 SC: Oh yeah. Okay. 0:36:04.0 ET: Really faking things. [laughter] 0:36:06.9 ET: And so I used to believe very strongly, I did quantitative political economy, election predictions and all that kind of stuff. And it's... I believed in it more I was doing it. And thankfully, most of my papers got replicated. [laughter] 0:36:26.9 ET: I didn't cheat that much. But the fact that it's medical research where there's so much, there are lives at stake. And the fact that in this country, it's the only place in the world where tremendous amounts of money are made from sick people. And from fudged data. And the FDA doesn't do all that well. And they fight a losing battle because anybody who is a good biostatistician is gonna work for a drug company. And it's a great big, it's an enormous, it's a public health problem. 0:37:04.9 S12: Yeah. 0:37:05.2 ET: The cheating, cheating in medical research is a public health problem and should be treated like a public health problem. And I wouldn't have said that 10 years ago or from all my other stuff. I was much more thought that numbers would help bring us the truth. And that's true. It's easy to, although the, it's easy to lie with numbers, but it's even easier to lie with words. [laughter] 0:37:31.3 ET: There's still... 0:37:33.0 SC: That's a good quote. I like that one. 0:37:34.4 ET: Like you can see the lies better than with words. But that really have crossed my skepticism strongly about the routine falsity of a lot of research on human beings. 0:37:52.5 SC: You did a very in-depth analysis that I learned a lot from, of the space shuttle disasters. And there I would say that it wasn't greed or an attempt to lie, but just people went a little astray in how they presented information and in it with terrible consequences. Could you explain, like to the people who don't know about this example what went wrong there? 0:38:18.5 ET: Yeah. I worked in the third book I have The Challenger, which was I basically have Richard Feynman's take on Challenger. And, I wish I was that sharper. He was really something and he tells a great story about it in one of the, you know... 0:38:38.6 SC: Yeah. 0:38:39.1 ET: But my own independent work was on... The second accident of the Challenger and... 0:38:52.2 SC: Columbia. 0:38:54.1 ET: And I got right after it went down, I got all the sly. So it was injured when at launch, when a piece of foam broke and hit the wing on the launch and made a hole size of basketball, and it flew for two weeks with a hole in its wing because there's no air up there. 0:39:21.0 SC: Who cares? Yeah. 0:39:22.3 ET: Who cares? Yeah. And they knew there was a problem. They may have known quite well by... Nobody ever says, but it may have been that a spy plane took a picture of that wing. But they don't want to express the resolution, you know, say what the resolution in the system. 0:39:39.1 SC: Sure. 0:39:40.2 ET: But they knew there was a problem and they did a engineers on about the fifth day, did a big PowerPoint presentation. I got that about a week after the Columbia went down. I got it via a federal government information thing the reporter had done and I got the slides. And I don't know anything about rocket science, but I know a lot about the relationship between evidence and conclusions. And that wasn't... What they were doing was perfectly clear to spot. And I take it with a key slide apart and you know they're in trouble because they're measuring things in cubic inches. [laughter] 0:40:30.3 ET: They're in trouble right there. There's a fam [0:40:32.7] ____ that two things crashed into Mars because of... 0:40:36.2 SC: Right. 0:40:37.4 ET: Nine, if that's straight there. But these guys were in cubic inches, but on one slide, they abbreviated cubic inches three different ways. 0:40:43.9 SC: Wow. 0:40:45.3 ET: If they were sophomores at MIT and a graduate student was grading their paper, graduate student would write on the thing, "Have you thought about insurance sales as a career? [laughter] 0:41:03.7 ET: Because that it's just a sign of something's really wrong here. That they can't get units of measurement right, and three different ways of writing it on one page. And none of them were inches cubed. They were in CU and just amateur. And they have some models and they tested. And so, but it all gives it away right there that they don't have the material to make a decision that everything is okay that's what they said. Everything's okay. And it was right there if you saw it... Especially if you knew it was an accident. But it's absolutely. So I did that on my own. I got a hit call from Boeing... [laughter] 0:41:57.9 ET: And that they had trademark on this, or copyright or something. And I said, "Well, it was gotten by a government thing, so." And the commission that investigated it published in their final report, which is the 100 page summary of all this stuff my analysis to that slide, and it said, we've gotta stop engineering by PowerPoint. And it turns out all the documentation of every project at NASA is done in PowerPoint slides. There's no technical reports. They used to do beautiful technical reports. They were famous for them. 0:42:34.3 SC: Sure. 0:42:35.3 ET: You know, like 10 pagers and stuff. And so they use it as a general attack on engineering by PowerPoint. But the point for me is that they, the shuttle people investigating who were a pretty fancy group investigating the accident, picked up on what I said. And I don't know anything about rocket science, but I know what the hell... I have a more powerful skill, that I can tell the difference between the relationship... I know about... I can understand the relationship between evidence and conclusion. And that's a different thing than knowing rocket science. [laughter] 0:43:13.9 ET: Because it's more general. 0:43:14.6 SC: Yeah. 0:43:15.3 ET: Broader. 0:43:16.0 SC: Well, it's very interesting because I know that, you've said you've criticized PowerPoint before very trenchantly. And I'll confess, I use PowerPoint all the time when I give talks. But I guess I would never think to use it to sort of share information as text. That makes no sense. 0:43:36.6 ET: As documents. Yeah. 0:43:37.4 SC: Yeah. 0:43:38.7 ET: Yeah. I had a wonderful thing happen very early on about PowerPoint, which I didn't hear about until 15 years later. So the PowerPoint essay comes out and it has a Columbia [0:43:51.5] ____ among other things. And it was called The Cognitive Style of PowerPoint. And it had a picture of Stalin giving a talk. [laughter] 0:44:00.9 ET: It has... And the people down below, of all the soldiers waiting down below, making remarks about his bad PowerPoint. And somebody read that essay before it was published and said, "Why don't you say what you should do instead of just saying bad PowerPoint?" And first, I got kind of stung. I said, "My job's not to rescue PowerPoint." [chuckle] 0:44:22.5 ET: And I thought, well, but I should rescue my audience, my people. And so I wrote one page in which is every meeting should begin with a handout that uses sentences, two to six pages using sentences, no bullets. And every meeting begins with a 30 minute reading period. And it's how I taught the confused Ivy League undergraduates, about we actually had to think in class. 0:44:54.1 SC: In class. Oh my goodness. 0:44:55.5 ET: In class, we had to think. That was, yes, they were used to scribbling, they were very good at scribbling notes. But they had... And so if I had a proof on the board, I would pass the proof out so they didn't have to take notes and then they can annotate how I got from step three to step four. Mark that up if they... And so they weren't in doing stenography anymore. So a few months after that came out, Jeff Bezos and his direct assistant were flying, reading aloud my essay in PowerPoint. And they saw that set of sentences, six pages, and they immediately adopted it. 0:45:36.9 SC: Okay. 0:45:37.2 ET: They threw out PowerPoint and the highest level of decision making was made by the... There was no presentation. There was no rehearsal or no slide. There was the six pager. People can read that out faster than you can talk. And so I have my courses, people got all five books, and there was reading all during the class. I mean talk a lot too, but we would stop and read these two pages and then I'd talk about it and so on. And so it was back and forth of them reading different mode and they could read... They could skip things they're not interested in. See with slides, everybody has the same slide at once, and it's controlled by the... But with reading, everybody in the room can use their own priorities and their own sense of relevance with that. And so all the work of preparation went into the report and a team might even do that six pager. Then they discuss it for an hour and a half. And he says, it gave us an enormous competitive advantage. And he said they wrote about this 15 years later. It gave us some tremendous competitive advantage, an order of magnitude, competitive advantage. They said this method, that everything that we want on went through this method and they just went batshit... [laughter] 0:46:58.6 ET: Over this. I didn't know about... I was feeling bullied by Microsoft and Boeing and stuff, and people saying bad things about me and stuff. And here Jeff Bezos was... The Amazon was doing this and thinking it was the greatest thing in the world. It was unbelievable. And I would've felt very comfortable and not so paranoid and threatened... [laughter] 0:47:21.2 ET: If I'd known, I could have told that story, but it didn't come out... 0:47:24.6 SC: Or maybe they could have slid you 1% of the profits. [laughter] 0:47:28.4 ET: Well, that's another point, which is I patented a medical interface a long, long time ago, but I decided to make everything open source. I thought about sparklines. Doing that and I had a patent lawyer who said they could do anything. They could patent anything. Microsoft patented Sparklines, but they didn't... They. 0:47:50.8 SC: Oh. 0:47:52.5 ET: And they use it as a trademark. [laughter] 0:47:53.9 ET: Stole it. But my view is I'm open source. 0:47:58.0 SC: Yeah. 0:47:58.7 ET: And also I'm doing just fine on all these books and all this teaching. 0:48:04.3 SC: Sure. 0:48:04.9 ET: I don't need... I don't like that. I like the idea that I'm so happy to get the ideas out. 0:48:09.9 SC: Yeah. 0:48:10.3 ET: And so hearing that Amazon was using this to great success and lots of other people that made me so happy but it had consequences. 0:48:21.6 SC: And the thing about meetings, this goes a little bit beyond the visual display of information, right? This is a kind of a way of thinking, right? And the PowerPoint critique gets into that as well. Is there a future book that has nothing to do with data visualization? [laughter] 0:48:41.2 ET: Well, you just exposed that I have a volume six, and... 0:48:48.7 SC: Don't mean to give it away. 0:48:50.2 ET: Let's see if I can tell you the title. Presenting analyzing data/information. Smarter communications, shorter meetings, content, credibility, clarity, efficiency, honesty. 0:49:08.7 SC: Oh, very good. 0:49:10.2 ET: Part two, how to evaluate presentations. How to make presentations so as both from the consumer and the producer point of view. See, so they're both thinking about credibility and content, consumer and producer. And then the second piece of this two chapter thing is data analysis, visualization, and the truth. And this is a sort of short course in ET. 0:49:39.0 SC: Okay. 0:49:40.3 ET: Those titles report, it turns out presentation means kind of everything. You could say it's a medical report. You could tell it's... But it's focused on both the production and consumption and the interplay between the producer and consumer. 0:49:55.5 SC: Well, it's crucially important in the modern world, right? There's so much... 0:49:58.1 ET: Yes. 0:50:00.5 SC: Information, so much content. Our attention is so very important. And there has to be huge inefficiencies that we haven't quite figured out yet. I know when I go hear someone give a lecture, seminar, colloquium in physics and it's an hour long, I will absolutely learn more from talking to them one-on-one for five or 10 minutes than I will from an hour long presentation. And... 0:50:23.0 ET: It's because you can control. Yes. You can find out... 0:50:26.0 SC: I guess so. 0:50:26.4 ET: What's relevant. 0:50:27.4 SC: I guess so. But is there... How do you make that scale? I don't always have access to the presenter myself for even 10 minutes. 0:50:38.4 ET: I write about efficiency. It's in the title. And, one of the principles is to show up early and finish early. The presenter. Show up early, chat people up, get them to start reading the documents and finish early. They'll be thrilled. No one ever wished the meeting longer. Thrilled. [laughter] 0:51:04.3 ET: The other thing is a good six pager can pretty well do it. I have thought times of, "Hey, read this and just say a little bit." And for my class, I do that. It's all there. Read it. They can read twice as fast, at least as you can talk. And they can choose what they read instead of the damn bullets. They can choose what's important to them. And everybody's reading differently that... It's a tremendous advantage to instead of poking through the slides too fast, too slow, too boring, they can skip paragraphs, they can go, "Oh, you're slower than we want." They can mark it up. They can read it twice. They can check something back. They can mark it up and then, oh, there's still a few minutes. Let me see now. This was good. This is good. I don't have to worry about that. And you could at some places, say anybody, wanna discuss this? People won't dare put their hand out. [laughter] 0:52:08.9 ET: No. But... Except people who always... 0:52:12.5 SC: There are some who always do. Yeah. 0:52:13.6 ET: Yeah. I had a strategy about people who ask questions, which is sort of this. You think about the priorities of all the other people in the audience, the person they'd most like to hear talk are themselves. 0:52:33.0 SC: I Agree. 0:52:33.5 ET: Secondly, they'd like to hear me. [laughter] 0:52:37.7 ET: The people they'd least like to hear talk are their fellow audience members. So on that principle, I decide I'll talk, you can handle the questions in maybe under 20 people. It's more of a discussion like in a little classroom or maybe 25. But it gets... When you... If you get over 30, you have now have people sitting in front who are people who ask questions at everything, they're professional. And it's often they have some other cause and... Or it's obvious or it doesn't... And we're wasting the time of the n minus one people in the room, especially if n is big. If there's 300 people in a lecture, you just can't do it. It's not a discussion. It's these little mini speeches often, or they sort of maybe one in 10 has some kind of grievance. Say... I've used PowerPoint for 20 years... [laughter] 0:53:47.3 ET: That strategy I just mentioned is in this essay about the audience and questions and about the priorities of the audience. And I think that's a very convincing argument about priorities of your audience. Who would you like to hear talking? You'll see, you can't talk yourself, they wanna hear the presenter. 0:54:04.2 SC: Yeah. It's a very big question for the modern world. How we share information and how we choose and what our time scales are and our methods. And I don't think that we're very good at it yet, so I'm glad to see you pushing to at least try to get better in that direction. 0:54:19.7 ET: You've got to because there's so much stuff. You've got... 0:54:21.8 SC: So much stuff. 0:54:24.2 ET: You've gotta... The other thing is there's some good research on ignoring things. 0:54:28.0 SC: Oh, yeah. [laughter] 0:54:28.2 ET: And being self-aware of ignoring and being self... Of deliberately ignoring this. I wish I could do that when I look at Twitter. [laughter] 0:54:39.1 SC: You're not alone. 0:54:42.1 ET: It takes away the whole afternoon... Not that word. Don't say it. I can throw away a whole afternoon poking around and it's kind of interesting. It's just interesting enough. 0:54:51.5 SC: Just interesting enough. 0:54:52.5 ET: Oh, how are the warriors doing? [laughter] 0:54:55.2 ET: They have a game tonight. I know. How are they? And it's requires... It's often beyond my self discipline. 0:55:04.2 SC: Yeah. 0:55:04.3 ET: I charge it away from my bed. 0:55:06.6 SC: Okay, good. 0:55:08.7 ET: I put my phone in The kitchen. It helps... 0:55:10.4 SC: We have to trick ourselves. 0:55:12.2 ET: Yeah. [laughter] 0:55:14.8 SC: Well, and the relationship between art and science here is fascinating because as a scientist, I like to think that theoretical physicist, we're trying to construct a theory, a model. We're working toward that. But as an artist, it's more of a craft. There's rules for this or suggestions or whatever. And I'm wondering if you think of your own work in visualizing information as being articulations of a single underlying theory or as just learning in a more piecemeal way and trying to use our judgment along the way. 0:55:49.3 ET: The smart aleck answer and half true is that inspiration is for amateurs and the rest of us just go to the studio every day and go to work. [laughter] 0:56:03.0 ET: And I see my art as... Well, what it has in common with real science is intense seeing and understanding that intensity brings some understanding and then ability to change and work from it, and so on. And everything serious requires, unless you're supremely talented like Picasso, requires a series of steps, of change, of discovering the material in art as you have an idea and I can tell within about half an hour whether this is gonna work out or not and I have learned if I can't find promise for the half an hour, is I call my backhoe operator in and say, "Bury that 'cause it's not gonna work. It's gonna waste time that I could... " I know this is gonna work. It's not working now, but I know it will. And then it's trying to avoid words of not saying, it looks like Picasso, it looks like Stonehenge. It's a piece of stoned. And so I see it for its traits not what it's name, but its color, its texture, and the very special thing about sculpture and about physics is air is a material. [laughter] 0:57:42.6 SC: Space. Absolutely. 0:57:45.0 ET: The only two things in that world are sculptures, air is a material and space is a material for in physics. 0:57:54.0 SC: Yeah. 0:57:57.4 SC: And that's, I'm not quite sure what that means, but, well it is thinking about air as a material. That's what sculptors do. When I'm working with my colleagues on a piece, probably half our comments are about airspace and it's like figure ground on paper that's, trivial. That's just flat land's design. That's I wrote a whole book about scape called this basically Escaping Flatland. 0:58:24.3 ET: And that's when I knew I was, graphic design was easy 'cause it was flatland [laughter] really hard stuff is you've got air as a material and it's vague and it changes from every perspective on every kind of light and every kind of sky and you know, and, and whether it's raining or not. And so there's such a multiplicity of things out there to sort among and make decisions about, but it's, I don't call it, it's not craft, it's a kind of the technical name of doing a piece is disjointed incrementalism. Okay. Otherwise known as muddling through [laughter] 0:59:08.2 ET: They're small steps. They're kind of separate steps... It is muddling through. It's trying things out. It has a clear stopping point... In the work I find it at the end of the first kind of ending the end of maybe a day or couple of days of work. I think it's really great. And I believe that fairly strongly. And I get up the next morning and see three things wrong and I'm willing to change my mind, but there's something I have to believe in it... For a while and defend it, you know? 0:59:46.4 SC: Yeah. 0:59:46.7 ET: And then gather myself with fresh eyes sometimes the next morning or maybe even the next day. And we said, we gotta do this now and I try that. And that's especially the case because I locate the art that is... I have 234 acres of my own... It shows only my sculpture. 1:00:12.7 ET: And we have backhoes and we make ridges. And no other artist has control of space of the land. Richard, Sarah gets a few hundred to maybe 500 square feet in front of a building laid out by an architect, and they say, that's where you're gonna put your piece. And this, I have control of the landscape of the trees, of there are seven ridges, ups and downs. The location, we can change it. 1:00:39.9 ET: I usually will change it maybe after six months to some better space. And all of that is now completely changing the air too. There is a material. And so it's just like in self publishing my books I have complete control. There's no bureaucracy, there's no... Thanks to the books I've been able to pay for all the sculpture stuff. 1:01:05.7 SC: Yeah. 1:01:08.0 ET: I do big landscape pieces, they're... The market for landscape, abstract landscape kind of art is like to market for Canadian experimental poetry... It's not a big market. Which is actually good because I'm free of pitching to rich people about why they should do it. 1:01:29.1 SC: Good. 1:01:30.6 ET: So that's the same thing with making the books. There's no bureaucracy, there's no editor. There's no editorial board, there's no... I choose the printer I work directly with. So it's all this hands-on thing that is combined with the mind. And those are, those two. It's those two things that have really helped me because though... There are no middle people. 1:02:00.9 SC: Yeah. 1:02:01.0 ET: There are no assistant deans. Universities have become bizarre in the last 15, 20 years with the deputy assistant provost and endless of it they... I have a law of university growth, which is the doubling time of the bureaucracy is 12 years... During which time the number of faculty and students remains constant. It's astonishing the rate of that. 1:02:26.5 SC: Yeah. 1:02:26.8 ET: Well, the doubly time that's it. 1:02:27.7 SC: It's something. Yeah, that's right. I do wanna give you a chance to talk about one other artistic thing, which is the Feynman diagram. 1:02:36.7 ET: Oh, yes... 1:02:38.0 SC: I'd love, all the Feynman diagrams in your new book. And the audience can't see, but as we're talking to each other, there are Feynman Diagrams behind you. Well, what is it that make, I love Feynman diagrams, but why do you love them as much? 1:02:50.9 ET: I grew up in Beverly Hills, or where I went to high school and I never knew Feynman. I never went to a talk. But a couple of my friends went to Caltech. I found out about his physics textbook, the three volume [1:03:07.9] ____. 1:03:08.8 SC: Yeah. 1:03:10.6 ET: And so that was kind of the introduction. And he has a very, the QED book is very good. It's mainly in English. It's four lectures he gave. And there are lots of Feynman diagrams and there are a miracle about you get the 17th significant digit accurate in theory and practice and so on. And so that went in the green, the fourth book law of refinement. And then here's how I started making the sculptures. I made a huge rocket, 80 feet long with an Airstream trailer on the end. 1:03:49.9 ET: And it's like in launch position only. It's at an angle... It's the sculpture of mine. Big sculpture, rocket science three. And it has lights inside and rotating TV antenna in an Airstream trailer with wheels and all this. And it's quite prankish. And I showed it in front of Family Lab along with Feynman's Van. I paid $10,000 to have Feynman vans repaired. 1:04:15.1 ET: They brought it in and they brought just my Airstream trailer in which I didn't ring the rocket in... That's 80 feet long and put it in front of Family Lab when... I believe they had a big show. Thanks. And the department of energy... Cabinet member came and loved them. 1:04:32.1 ET: And so they bought a bunch of them... And they're at Family Lab. I first used them on that Airstream trailer. It's going to some other place. And the aliens or the people... Whoever they are far away, they're not gonna understand what the NASA logo was or the flag. They will understand Feynman diagrams... 1:04:58.4 SC: Interesting. 1:04:58.9 ET: And they'll say the people inside are pretty smart, because it's universal. 1:05:05.3 SC: It's universal. 1:05:07.2 ET: And the... Well we find the diagrams all kinds of places because they're based on nature's laws. And some they'll see... They'll see a little code of it and so on. And so it was a lovely prank and I just put three Feynman diagrams on the side and they cast shadows on the aluminum thing and it looked beautiful. 1:05:28.8 SC: They're also beautiful. That's the thing. Right. 1:05:31.4 ET: And this turned out to be the G minus two or something, thing... Came out just a year or two ago. That was an accident. I chose these because they were 10th order and they were the coolest ones. And a guy had printed like 500 of them. There's thousands of dollars... But these are, I believe 10th order Feynman diagrams. And it happened to come up with the... It was minus two. 1:05:57.5 SC: G minus two is the new one. Yeah a way to find new physics, right. 1:06:02.1 ET: Yeah. Yeah. Yes. And I showed that at Family Lab about three or four years ago. And that was kind of scary because they actually knew what these were. People understood them. And one of them said... I had complained about one of the things that was maybe not appropriate if you know. He said, "What's that?" And I had prepared, and I had said, "Well, you guys divide by zero." [laughter] 1:06:37.2 SC: And but we divide zero by zero so it's... Maybe it's okay. [laughter] 1:06:43.3 ET: Maybe it's a professional who said okay. And then Feynman hundredth birthday, the Nobel Foundation took them on... Or the Nobel Library took them on some tour. They went to Hong Kong and then COVID came. Went for his hundredth anniversary assignment. And that was... I didn't go to the opening unfortunately. It was just too much to do. I'd already seen him. But the pictures are wonderful and that just made me so happy. They were out there and the secret of them is to make them, raise them about two to three inches off the surface. Then they cast shadows. And they're the most amazing things because you have a couple of perpendicular things coming out that pull them away, which create now a space going backwards to the wall from the metals to the wall by those posts. 1:07:50.2 ET: And the shadows look three-dimensional because of the light. And then of course the shadows change as the sun goes by. And so they just are totally alive and always different because the shadows are strong often and along with the silvery part. And this is his sign of a really good art piece with, where he said, you get all kinds of amazing things for free. That's my favorite expression about a piece of art. It's not true of all, that is things you didn't plan on at all, but when it was raining, it did something magical. Or when there was butter light with sunrise and sunset, it did something magical with the butter light of sunrise and sunset. And I call that this is all for free. And that's such a wonderful thing to make a piece that you can say that about. You can say that about a lot of outdoor pieces cause you've got rain and shadow and the light and the earth rotating, all of that. And all those were for free and completely unfathomable until you actually see them, the combination of a shadow, but it's also raining and or dogs running by and there's this and that, and that's a way of a good piece of art. And you never say that against on a flat on the land. 1:09:19.4 ET: It's the dead stain, the optical experience. You may be able to see different things somewhat, but not this stuff you get for free every day you see us so this gives you free fresh eyes. I have a flat painting say on the inside. After a month, I don't even see it anymore I have to move it. These things, the pieces out here are... They're always alive, they're always different. I'm always happy to see them. Yeah. And I like rain the best, the rain makes the rust and the iron in the stone may have this nice warm moisture luscious color to them. Plus the stone looks great when it's wet. 1:10:03.7 SC: Well, I mean this forces me to ask one more thing then we can wrap it up, which is the importance of the time domain to everything that you're talking about. You're making the point that these kinds of works of art, even though they're stationary, they interact with their environment in such a way that at different times, they give you different experiences. But you've also really emphasized even in the data visualization world, the importance of when you collect the data and the importance of when you present the data. And maybe this is yet another frontier that people need to think about. 1:10:38.1 ET: I think the best idea in the new book is the following. You never learn more about a process than when you watched the original measurements being made. [laughter] 1:10:53.1 ET: That came from a great applied statistician with a lovely name Cuthbert Daniel, who did industrial statistics. And so, add a little more sulfur and step 34 that's Indus... And he do little experiments and was incredibly smart, MIT, et cetera. And he did applied work and made it. He was a famous consultant for on drugs, but on industrial processes. And we wrote a paper together about the FDA. And he told me that, and he said, "Here's an example. General Electric put PCVs in the Hudson River. And they spent billions trying to make up for it. And they had to do testing every day of how... How good the water was and report to the EPA. And so let's go out in the little boat to make a test." And it's a guy in a little small boat and he's got kind of a cup on the end of a stick. And he leans over out of the boat to take a sample and he leans over to the side where the water's cleaner. The statisticians maliciously call that the sampling to please. [chuckle] 1:12:15.7 ET: The only time that you can see that is when you're watching the measurement. Right. You're there, and I have come to play on many people's backyards. I watched three surgeries at the Cleveland Clinic. One open heart, two robotic. I've solved NASA's toys play at other people's backyard. And I did a section in about being at the point of measurement in the book and I did an interview about once every six weeks with a ICU and emergency room nurse for four years before... Five years now before the epidemic because she was my air dresser. For an hour and a half I would just ask her a question and another question and just listening prompting and see when there's... When the system variance is zero. You can use it... You need an end of one [laughter] That's a cool point. So she was talking about drawing blood. She's talking about all drawing blood. 1:13:31.5 ET: There isn't much a variation. You're gonna hear the most everything in this one sample of size one. That's a very good way and it's accidental. She's come to me. It's not any kind of drawing... It's not any kind of cherry picking. It's just convenient. I learned so much from her of the general system of how it worked. I'm not there in the ICU wither her, but I am asking questions emergency room now. And I know you think, this is happening in every emergency room in the country. They're totally jammed up and these things happen and they're aways filled with their needs and stuff. 1:14:17.8 ET: It's better than chosen anecdotes, because we're having this observer who's just telling us what's going on. That I think's the best idea in the whole book and it's in my spirit of being hands-on. Instead of these guys writing the 500th way to do such and such logistic regression is what everybody uses. Go out and watch how this stuff's measurement... Get out on the field and watch the measurements and your eye... The scales will fall from your eyes. You're assuming these observations are independent. [laughter] 1:14:57.7 SC: There's a saying in physics that "No one believes a theory more than the theorist who proposed it and no one is more skeptical of data than the experimentalist who collected it." [laughter] 1:15:06.7 ET: I like that... I like that 'cause they know the dirty stuff [laughter] 1:15:11.6 SC: Exactly. I mean, even that's probably the most important reason why we do labs in undergraduate physics. So you know that it's not a completely clean painless experience to collect that data. There's some things that happen that you should be aware of. 1:15:25.3 ET: It's because you often... A person doing it often knows the right answer. That's the... You're seeing measurement error right from the start in a way. 1:15:41.8 SC: Yep. Well, Edward Tufte, thanks so much. I think you've given us a lot to think about and new ways of thinking in a very information rich world that we live in today. 1:15:49.6 ET: Well thank you. I've loved doing this. Good. 1:15:51.3 SC: Thanks. Bye-bye. 1:15:52.3 ET: Good. Take care. Bye.