Via JenLuc Piquant’s twitter feed, here’s one time I’m not going to stick up for my colleagues in the social sciences: a misguided attempt to cast the search for empirical support as “physics envy.” It’s a New York Times Op-Ed by Kevin Clarke and David Primo, political scientists at the University of Rochester.
There is something rightly labeled “physics envy,” and it is a temptation justly to be resisted: the preference for reducing everything to simple and clean quantitative models whether or not they provide accurate representations of the phenomena under study. The great thing about physics is that we study systems that are so simple that it’s quite useful to invoke highly idealized models, from which fairly accurate quantitative predictions can be extracted. The messy real world of the social sciences doesn’t always give us that luxury. The envy becomes pernicious when we attack a social-science problem by picking a few simple assumptions, and then acting like those assumptions are reality just because the model is so pretty.
However, that’s not what Clarke and Primo are warning against. Their aim is at something altogether different: the idea that theories should be tested empirically! They write,
Many social scientists contend that science has a method, and if you want to be scientific, you should adopt it. The method requires you to devise a theoretical model, deduce a testable hypothesis from the model and then test the hypothesis against the world…
But we believe that this way of thinking is badly mistaken and detrimental to social research. For the sake of everyone who stands to gain from a better knowledge of politics, economics and society, the social sciences need to overcome their inferiority complex, reject hypothetico-deductivism and embrace the fact that they are mature disciplines with no need to emulate other sciences…
Unfortunately, the belief that every theory must have its empirical support (and vice versa) now constrains the kinds of social science projects that are undertaken, alters the trajectory of academic careers and drives graduate training. Rather than attempt to imitate the hard sciences, social scientists would be better off doing what they do best: thinking deeply about what prompts human beings to behave the way they do.
Sorry, but “thinking deeply” doesn’t cut it. People are not especially logical creatures, and we’re just not smart enough to gain true knowledge about the world by the power of reason alone. That’s why empiricism was invented in the first place, and why it’s been so spectacularly successful over the last few centuries.
Clarke and Primo seem to confuse “the need for empirical testing” with “the need for every model proposed to be backed up by data before it gets published.” If they had stuck to rejecting the latter narrow idea, they would have had a decent case. Certainly we physicists don’t require that every model be supported by data before it is published — otherwise my CV (and those of most of my friends) would be a lot shorter! But we all agree that the ultimate test of an idea is a confrontation with data, even if a theory might be too immature for that confrontation to take place just yet.
In reply to Tyle Stelzig (#19):
I read Clarke and Primo to mean that anything that doesn’t fit the “hypothetico-deductivism” approach will not be published, funded or even thought about. Without Clarke and Primo here, I suppose we cannot get clarification on that.
I’m not sure I understand Chalabi’s point entirely. I don’t think economics suffers from an over-reliance on math necessarily (though that may be a symptom of the disease). I think it suffers from a pervasive ethos of prescription instead of description, and if the world doesn’t conform to some-or-other beautiful model, well, so much worse for the world, then.
Clarke and Primo have a book “A Model Discipline: Political Science and the Logic of Representations”.
http://www.amazon.com/Model-Discipline-Political-Science-Representations/dp/019538220X
The publisher’s blurb reads:
“In A Model Discipline, Kevin A. Clarke and David M. Primo turn a critical eye to the methodological approach that dominates modern political science. Clarke and Primo contend that the field’s emphasis on model testing has led to a distortion of both the modeling process and the art of data analysis and cannot be logically justified. The authors argue that models should be seen as “objects” and thus regarded as neither true nor false. Models should instead be evaluated for their usefulness for a particular purpose. Divided into two parts, the book first establishes that current practice is not philosophy-free and rests on a number of questionable assumptions. The second part focuses on the different ways that theoretical and statistical models can be useful, and closes with a defensible justification for integrating theoretical and statistical models. A novel work of methodology, A Model Discipline offers a new perspective on long-held assumptions about the way research in the social sciences should be conducted.”
There is a review by a reader, worth reading:
http://www.amazon.com/review/R29A0JZ43PU7EO/ref=cm_cr_dp_perm?ie=UTF8&ASIN=019538220X&nodeID=283155&tag=&linkCode=
A quote from that review: “Clarke and Primo tackle head-on a number of thorny philosophy-of-science questions that are usually ignored by political scientists and persuasively argue that the community of political scientists has committed itself to an untenable understanding of models and their purposes. Rather than merely criticizing, they offer a viable alternative to the hypothesis-testing that now dominates the field: namely, we should think of models as objects like maps, and evaluate them not as to their truth (since objects are neither true nor false) but as to their usefulness.”
“On their discussion of usefulness, Primo and Clarke earn two cheers. In Chapter Four, they explain how theoretical models could play a variety of constructive roles: as foundations for more models, as a way of organizing existing empirical generalizations, as a means of investigating causal mechanisms, or as devices for forecasting (which they note are relatively rare in political science outside of election studies). All of this is helpful–but I was somewhat disappointed that in their exploration of the purposes these models can play, the authors almost entirely eschew the question: “useful to whom?””
End quote.
The first few pages of the book are also available, and yes, it sounds like Primo and Clarke argue that the analog in political science of the Chern-Simons model in physics would be mishandled in current political science practice.
http://www.hss.cmu.edu/philosophy/glymour/glymour1980.pdf
Hypothetico-Deductivism is Hopeless
Clark Glymour
Philosophy of Science, Vol 47, No 2 (Jun 1980) 322-325
Ouch, and that leads further into the morass:
The Raven Paradox
http://en.wikipedia.org/wiki/Raven_paradox
Objects are dealt with through their definitions. If arbitrary, as part of an axiomatic deductive system, they are assumed to be ‘true’.
More typically, definitions of objects represent shared descriptions that fit fairly well with the empirical observations made within a discipline. In this case, they are used with a grain of salt, because of the inevitable residual uncertainty related to all inductive reasoning about observables, and this is just another face of “no scientific model can be either absolutely falsified – or absolutely proven.”
The Statement “objects are neither true nor false”, therefore is thoroughly misleading with respect to this discussion.
The source of much of the confusion among some scientists, and most others, in deciding where scientific ‘facts’ and models belong within the spectrum of beliefs, follows from too common, glib ASSERTIONS about ’empirical truths’ associated with the most well established ‘theories’ and ‘laws’ of science. Such ‘truths’ are always subject to possible (even probable) revision.
Some economists and politicians – without understanding that even these scientific gems are at best, provisional – may ‘envy’ science because they wish they could make such assertions within their disciplines with equal confidence. They don’t appreciate that unqualified assertions of scientific ‘truths’ are hardly distinguishable from some assertions of religious dogma.
“Rather than attempt to imitate the hard sciences, social scientists would be better off doing what they do best: thinking deeply about what prompts human beings to behave the way they do.”
aka religion and we know how well that works considering that one religion Christianity, has some 10’s of thousands of sects, each which claims to have thought “deeply” and who claim to have the only ‘right’ answer.
Re my post number 25: I apologise for apparently somehow completely missing Sean’s final paragraph. If I’d seen it before I posted I wouldn’t have bothered.
I absolutely love this post.
Also…does this mean Sean thinks string theory is not a real scientific theory?
I don’t share Noah Smith’s love, but I share his question. And also, of course, dark matter. And all the other dark stuff. And anti-materia. We can go on.
I’m not sure which of the many versions of empiricism Sean subscribes to, but is clear that the model criticized in the op-ed, hypothetico-deductivism, has an abysmal track record in social science. I challenge Sean, or anyone, to mention one important discovery about the social world created by this method. And no, important discoveries about the physical world don’t count.
That is not to say that empirical matters don’t matter. But it is to say that social science needs to get rid of the physics envy.
Among empirically-minded linguists seeking to criticize less empirically-minded linguists, the phrase “physics envy” has a different meaning than the way Clarke and Primo use it. For us, “physics envy” is not taking a rigorous empirical model an applying it inappropriately to fuzzy human data. Instead it borrowing unrigorous mathematical ideas to cover up for the fact that you don’t want to build an empirical model. This cartoon sums it up nicely.
Highly empirical computational linguistics people like myself end up having more in common with the “soft science” branches of linguistics–the parts that bleed over into psychology and sociology–than the apparently “hard science” branch of syntax (i.e. the stuff Chomsky does). The working style for the former is often to perform experiments about human language (e.g. reaction time psychological experiments with linguistic stimuli, or sociological surveys of who has what accent). These are often empirically rigorous. (Where part of rigorous means adding the appropriate caveats about human behavior being a particularly difficult thing to quantify.) Many theories of syntax, on the other hand, look at first glance like mathematics papers–there are theorems and Greek letters all over the place, and the only data is anecdotal–but lack the axiomatic rigor to be called math.
So we all agree that “physics envy” is bad, but different practitioners may have different ideas of what makes it so.
Since the seven maintainers of this blog are all physicists and astrophysicists, and the subject of this thread is the importance of “Testing Your Theories,” I will try once more to evoke a response concerning a large area where physicists feign knowledge instead of testing their theories.
Ironically, but not surprisingly, psychological and sociological influences bear heavily on the persistence of this practice. It is not surprising because physicists are fallible human beings, not enviable gods. As such, physicists are proud of their profession and desire to cast it in a positive light. An expression of this “soft” underbelly of physics was conveyed by the physicist Daniel Kennefick in response to his experience and interactions with the community of gravitational wave theorists. Kennefick wrote:
“There is a preference not to remember or not to overstress the significance of something which may be seen as vaguely disreputable to the field. It is a characteristic aspect of physics that to pose a problem or a question may, in itself, be taken as a sign of bad character.”
Recall that the question that I’ve posed here and elsewhere (posts #17 and #23) concerns the result of testing the commonly discussed gravitational interior solution (Newtonian or relativistic). What happens when a test object is dropped into a hole through the center of a larger massive body?
We don’t know the answer because empirical evidence is completely lacking. “…‘Thinking deeply’ doesn’t cut it.” Nor does the “reflexive” response of deferring to the energy conservation law, because this is one area where energy conservation has not been tested. Carroll’s ostensible adherence to empiricism implies that he should be eager to resolve this matter with an experimental test.
So how do we explain the fact that he and other physicists choose to ignore this area of their ignorance? I think the answer resides in a parallel between Kennefick’s above characterization of the sociology of physics and the case of Galileo, when he was up against the Church. One of the latter’s key advocates was Roberto Bellarmino, the heretic hunter, who referred to those who would question scriptures as “dangerous and in some cases embarrassing to the Company.”
The standard Newtonian and relativistic interior solutions have never been tested, but they are as Gospel. To question them is a “sign of bad character.” It would be embarrassing (to the Company) for a physicist to admit the negligence of leaving this stone empirically unturned. So they carry on, only pretending to know what they would find if they would bother to look.
As “people,” Sean Carroll and his colleagues are not—by his own admission—“especially logical creatures.” Of course this is also true of myself. But in the present case, I am the one urging that we test our logic and our theories by inquiring of Nature. Whereas Carroll and his colleagues, by ignoring my posts, seem to be relinquishing their ideals, i.e., the need to test their theories.
Drop a test object into the hole. What happens? Theories and logic won’t help here. Professor Carroll, what happens? Why don’t we find out?
Benish: Physicists have access only to finite resources, and therefore have to choose which experiments are most likely to lead to surprises. The experiment you describe simply isn’t one of these – physicists are fairly confident it would go the way the expect, having done similar (but yes, not identical) experiments many times. So it isn’t worth the opportunity cost. Better to use the money looking at things we know less about.
But no one is stopping *you* from doing this experiment. If you don’t share physicists’ confidence that existing models of gravity would correctly describe your ball-in-hole scenario, then by all means, try it out! If you stumble upon a surprise, physicists will be grateful and excited.
Pingback: The Problem of Instructions | Cosmic Variance | Discover Magazine
Sean Carroll wrote:
Excellent point! But it is not needed to go to social sciences to discover that real world is not so simple as physicists believe. Precisely, the new encyclopedic article The hierarchical structure of matter contains a discussion of Feynman’s incorrect approach to the molecular structure of water.
As many physicists, Feynman believed that water is only composed of molecules of H2O. This is a common mistake that he shared among many of his readers. As any general chemistry textbook explains –in the chapter dealing with atomic and nuclear issues– molecules are, in a strict sense, not of the same sort. Instead they are built up of different isotopes of hydrogen and oxygen in various combinations (e.g. water also contains molecules of D2O), and such a difference is important to some chemical problems. A few chapters later, chemists learn that the H2O molecules are not inert, but dissociate into H2O+ and OH− ions, and that the concentration of all three components in water varies with temperature and pressure. The concentration of those ions governs the chemical properties of water. Another chapter that deals with electrical properties of the same pure water will explain to students that molecules are not isolated blocks but dynamically forming various entities through aggregation and disaggregation, governed by various forms of changing interactions, such as covalent bonding, hydrogen bonding, van der Waals interactions, or electrostatic and spin-spin interaction, depending on the level of molecular description. The atomic-molecular picture is still more complex, but the level of detail presented here is enough to understand that, contrary to physicists’ belief, water is not made only of H2O molecules.
@40. Juan Ramón González Álvarez:
A typo: “A few chapters later, chemists learn that the H2O molecules are not inert, but dissociate into H3O+ and OH− ions”.
@17. Benish:
Completely agree on “The media is barraged with science shows and articles about black holes and wormholes.” In my opinion, the major problem is that the academic literature on black holes is open to many fundamental objections. See last posts on
http://fqxi.org/community/articles/display/165
http://fqxi.org/community/forum/topic/1246
The section 8 of the paper “Non-redundant and natural variables definition of heat valid for open systems” (cited therein) corrects many nonsensical claims done in a recent review on black hole ‘thermodynamics’ by Wald.
Tyle Stelzig:
Physicists’ confidence in the standard prediction for the interior solution experiment can be seen as misplaced in various ways. Before stating a few of them I would urge that the ideals of science are sufficient reason to insist that the experiment be done. Confidence is no replacement for empirical evidence. Furthermore, the lack of resources argument is hard to swallow in light of the seven, eight, and nine digit budgets of gravity experiments like LIGO, LISA and others. The experiment I’ve proposed would cost a small fraction of these.
What we know about gravity is almost entirely based on observations over the surfaces of large gravitating bodies. On a graph that plots the maximum velocity due to gravity as between two bodies, the curve is reasonably complete from the surface to infinity. But from near the surface inward, we know virtually nothing. There is no “similar” experiment to the one I’ve proposed because we’ve never seen objects fall from near the surface of any large body to near its center.
If we think of the Earth as our primary source of empirical gravitational knowledge, then what we know derives essentially from the horizon upward. Failing to extend our knowledge inward, means in effect, that we have failed to explore the most ponderous half of the gravitational universe. Physicists’ confidence is based on mathematical extrapolation from the explored exterior to the unexplored interior. The late Herman Bondi poignantly expressed the serious inadequacy of such partial knowledge:
“It is a dangerous habit of the human mind to generalize and to extrapolate without noticing that it is doing so. The physicist should therefore attempt to counter this habit by unceasing vigilance in order to detect any such extrapolation. Most of the great advances in physics have been concerned with showing up the fallacy of such extrapolations, which were supposed to be so self-evident that they were not considered hypotheses. These extrapolations constitute a far greater danger to the progress of physics than so-called speculation.”
This of course echoes Carroll’s remarks about the need for empirical evidence to replace all-too-often fallible human logic. Lip service to the ideals of science is extremely common. The idea is simply to test what we think we already know and to look where we have not yet looked. The need to do this even for simple problems whose results are unthinkingly taken for granted is amplified in the present case because of our gross ignorance of the ultimate nature of gravity. Nobody has ever figured out what matter supposedly does to produce gravitational attraction. What does matter do to warp spacetime? In general relativity the standard prediction for the interior experiment is based on the assumption (Schwarzschild interior solution) that the rate of a clock at the center of the body is a minimum. What does matter do to make the central clock tick slow? Intuitively it makes more sense (by symmetry) that its rate should be a maximum (in which case the falling body would not pass the center).
Nobody has figured out how gravity relates to the other forces. Some physicists have the impression that after many decades of effort with virtually no progress, we are badly stuck. Given this situation, isn’t it obvious that the most ponderous half of the gravitational universe ought not to be presumed known before being checked?
Deeply ingrained acceptance of standard idea is the source of your confidence. Please consider the possibility that many of these ideas are quite wrong. Why not generate interest in doing this experiment if only to finally confirm that its result is not a big surprise?
Anybody notice that this was published on April 1st?
This Blog is indeed great!
For economics, people have been discussing these issues—that is whether economics is science, or not, plus some comparisons with physics, or other disciplines— for years I can say.
The econ literature has great discussions on that too, for example this article by economist Hal Varian from Berkeley:
http://people.ischool.berkeley.edu/~hal/people/hal/NYTimes/2002-08-29.html
or by Greg Mankiw from Harvard-econ:
http://gregmankiw.blogspot.com/2006/05/is-economics-science.html
also here
http://www.economics.harvard.edu/faculty/mankiw/files/Macroeconomist_as_Scientist.pdf
and the blog of S. Williamson from Washington University – econ
http://newmonetarism.blogspot.com/2011/09/ugghh.html
with a particular focus on financial markets or asset price dynamics I think I should say, here is a debate, a manifest, on “Physics vs Finance”:
http://www.ederman.com/new/docs/fmm.pdf
The list goes on..
Frankly speaking, I do not know the answer, yet, whether economics is science or not. I definitely disagree though by only looking at the dictionary, we can tell if a discipline is science. Dictionaries may give the definition of what science is but it cannot answer e.g. how do we obtain scientific knowledge? Why? What are the limits of the scientific knowledge, Should we take an empiricist way or positivist way? Subjective knowledge or objective knowledge? etc..
I think economics is a quite interesting discipline to study though: A part of it is applied math (e.g. game theory, e.g. Nash), a part of it is engineering (operations research problems such as kidney exchange, mechanism design, industrial organization, production planning), a part of is sociology, history, psychology (neuroeconomics, behavioral finance), etc…
Just for curiosity, Dr. Carroll, as a physicist, do you think economics is science? I took liberty to directly as you because in the economics research community I can say, there are physicists who think that economics is a science, and there are some others who think otherwise. I was just wondering what you think.
—
Mete
For those who might be interested in, another great discussion on “economics vs. physics is here:
http://noahpinionblog.blogspot.com/2012/03/scientific-failures-particle-physics-vs.html
—–
Mete
#42:
With the advent of standard cosmology after WMAP, general relativity has been successfully tested with precision (I think) and out to distances (I am certain) as never before. So no one entertains the notion that it can be wrong.
However, the theory may be incorrect; it is certainly known to have a domain of validity as it is an effective theory. The means to show it so is for you to do the experiment you envision. Especially if it would “would cost a small fraction” of other experiments that scientists deem important.
I think Clarke and Primo are mostly arguing against a trend in their own field (political science) where the major journals refuse to publish a theoretical model without some sort of empirical “test” of the model *in the same paper.* Never mind whether the test actually captures key aspects of the model or not or that the data may not be yet available to do a proper analysis.
I can’t believe that came out of Rochester. Theirs is one of the most famously empricist political science departments.
“People are not especially logical creatures, and we’re just not smart enough to gain true knowledge about the world by the power of reason alone.”
What about topology? Mathematics in general? String theory? Multiverses?
I think it is totally normal for all fields of sciences to sometimes have theories without being able to test them. Such theories can be extremely useful and stimulating for the field.
In social sciences, which partially concerns the workings of society as a whole, experiments are often impossible.
For example, someone might have the hypothesis that it would be good for society women should be allowed to vote. Others will argue that this will destabilize the society and cause problems.
Isn’t it completely fine to have this hypothesis, think deeply about it, and argue for and against it, without being able to test it at first? And I don’t think we have actually tested it. We decided to let women vote. The society is now what it is. We can’t compare to a case where we women are not voting.
I also think the hypothesis that they actually give in the article is very interesting, namely that competing political candidates will converge to similar views. I think this is a very useful theory to have when thinking about politics, but it will be quite hard to test.
Re: #46—Torbjorn Larsson,
Notice that this domain of validity is only for the region from near the surfaces of gravitating bodies, outward. The motion of bodies due to gravity in the domain from near the surfaces, inward is completely unexplored. If claims of the validity of general relativity—and even Newton’s theory—were more explicit (honest) about this, perhaps someone would be inspired to fill the gap.
It is the job of physicists to fill this domain with empirical evidence, not mine. Filling it with “deep thinking,” “confidence,” and exclusion of “the notion that [Einstein] could be wrong” is the same kind of sin that this thread so meekly holds against social scientists.
Physicists: Please test your theories. Back up your words with actions. (And don’t quibble about or make excuses regarding money. I’ve spent plenty trying to do the experiment myself in a makeshift laboratory. An institution grade laboratory is needed. But the cost would be in the 5-6 digit range, not the 7-9 digit range of big science.)
Finally, following my post #42, for a short while there was another post that briefly echoed a liking for what I wrote. Why was that post removed?