The Origins of Spurious Explanations (Part I)

Social scientists are supposed to provide explanations for events in the world, usually ones they find interesting.  But how often are such explanations spurious?

A ”spurious explanation” as I am defining it, is an explanation of an event or phenomenon that appears to be plausible, but is in fact false.  In other words, a spurious explanation has nothing to do with the direction of causation or the potential for confounding variables.  A spurious explanation occurs when the wrong reasons are given for explaining an event or the relationship between two variables.

There are undoubtedly dozens of reasons why spurious explanations occur and understanding their causes would help in avoiding them.  In each of the next two posts, I will take up one aspect of spurious explanations.  Here I will talk about spurious explanations for infrequent events.  Part II will talk about spurious explanations for trends.  The two parts will quite different, which likely reflects my general incoherence.

The most obvious characteristic of infrequent events is that there are few others like it.  Events such as financial crises’ and wars are to some extent unique (or if you don’t think they are unique, you can read part II when I get around to posting it).  As such, infrequent events are often crying out for an explanation because they stick out from the crowd of regular events.

But consider the following.  First, over a given time interval, a particular infrequent event will not have a zero probability of occurring.  Wars and financial crises are infrequent, but one cannot say that any particular war will not happen next year with certainty.  Second, now consider the entire set of potential infrequent events.  It is important here to realize that this set includes both events that have already happened and not happened yet.  Notice also that this set is also extremely large.  There are literally millions of potential events that could occur, although the probability of many will be very, very low.  Put these two together and it directly follows that explaining the causes of a particular infrequent event will be extremely difficult because it may have arisen purely due to chance.

Why chance?  It is because even though a particular unique event will have a very low probability, the probability that at least one of these infrequent events will occur can actually be pretty high.  Consider 100 (independent) unique potential events, each with a 0.001% chance of happening in a year.  What is the probability of observing one of these events in a given year?  Well, there is a 100 x 0.001% = 0.1% chance that any of these events will occur on a particular day and a 0.1% x 365 = 36.5% chance that any of these events will occur in a given year.  In other words, even though each event is extremely unlikely, you would expect to see one of them within a three years.

How does this translate into spurious explanations?  Say one of our 100 events occurred in year three.  An enterprising social scientist might say, “That’s interesting.  That event was so rare, probably only about a 0.001% chance it could happen.  Something must be going on here.”  Our budding social scientist would no doubt do her/his research and find out that perhaps, to her/his surprise, a few years before a policy change took place that fits into one of the theoretical frameworks she/he heard about at recently.  The social scientists would no doubt be excited that an explanation has been found.  However, unaware that it is dangerous to disregard the probabilities of unobserved events, there is a good chance that this explanation is spurious.

The Lucas Critique and Politics

The Lucas critique is a favourite concept of mine.  It says that it is naive to try to predict the effects policy changes based on relationships identified in historical data.  The reason it is naive is because identified past relationships are conditional on the environment of the time and since policy changes can affect both the variables of interest and the environment itself, expecting the same relationships is folly.

If one thinks about it as a concept, the critique should apply to many political dispositions too.  For instance, people on the economic right want to extend market freedoms as much as possible, claiming that since choice is good, more choice is better.  This may in fact be true.  It was certainly true in the past, more choice in a whole host of markets has led to increases in human welfare.

But can we use this evidence to say that markets (and hence choices) should be expanded even further?  Maybe, but since this argument is based on a past historical relationship (i.e. more economic freedom = more welfare), it would be naive to think that such a policy will always hold.  Afterall, the environment in which more freedom was extended in the past was one that contained both freedoms and restrictions.  today there are more and more freedoms and fewer and fewer restrictions, so the argument for expanding markets actually fails the Lucas critique because today’s environment is much freer than in the past.  Perhaps we have reached a point where the marginal benefit of extending economic freedom and markets is negative because additional externalities will pile up without offsetting benefits.

The exact same argument holds for the left as well.  I would argue that policies such as high rates of union membership may have proven beneficial in the past.  They helped usher in many of the benefits that we all take for granted today, such as the 8 hour work day and vacation time.  However, this is still insufficient evidence regarding whether or not higher union membership would be beneficial today.  Maybe unions were helpful when working conditions were atrocious, but now they simply drive up prices.  Just as with the right, if the environment changes, you can’t rely on past relationships to guide policy.

But the whole point about this is not that we shouldnt maximize freedom or support unions.  Indeed, i think we can do both.  The point is that past historical relationships can at best be a rough guide to future policies.  Even if we can identify these past relationships with precision, this is still insufficient evidence from which to conclusively guide future policy.

This was my long way of saying that context matters and we know far less than we think we do.

Tailoring Policy to Local Conditions

One always hears that tailoring policy to local conditions is a good idea.  But to me, its strange because the idea seems to be one of those that sounds uncontroversial until one goes to apply it.

Consider the following justification for favouring a ‘local’ approach to policy: how the heck can some bureaucrat X km away in capital city Z understand the local conditions and what this community really needs?  Who else knows what this community needs than the local residents?

At first blush, tailoring to local needs seems plausible because within any state, there will be a diverse array of communities and a one-size-fits-all approach will be inefficient if not downright destructive.  But if this is true, surely the same will hold true at more decentralized levels.  In other words, if nations are diverse and policy should be made more locally, states and provinces are probably equally as diverse, so making a one-size-fits-all policy for a state fails under the same logic.  If we keep going with this argument, it turns out that most cities are diverse places too and a one-size-fits-all approach will level fail this test too.  What about neighbourhoods?  Wait, those are diverse too.  What about streets?  Streets appear somewhat homogeneous (in terms of income, education, and often ethnicity), but who agrees with their neighbours about everything?

So if one follows the logic of wanting policy to be local, one had better know what particular ‘local’ they are talking about.  And if one wants to criticize policy on the grounds that it is insufficiently local, one had better have a good idea of the appropriate level of decentralization the policy should rest at and why it should go no further.  If you can’t answer that question, criticizing a policy as insufficiently local comes across as complaining, rather than articulating a well thought out alternative argument.

Can Democracy Ever Explain Anything?

What can democracy explain?  I am coming to think more and more that it explains very little. When I say it can explain very little, I mean it explains very little econometrically because democracy tends to be stable over time (at least in economically developed states).

 For a regression to have any hope of being valid, variables (or some transformation) must be at least mean reverting (i.e. the data must “regress” back to the long run average).  But democracy is no such variable.  There is no long run average level of democracy around which yearly observations fluctuate.  Thus, changes in variable X cannot be explained by changes in democracy because democracy doesn’t change.

There are only two source of variation in democracy I can think of: (1) with developing states as they move back and forth between more or less autocratic and more or less democratic regimes; and (2) cross country comparisons. However, both of these sources of variation have big problems.  For (1), it will likely prove very difficult to piece out the effect of democracy from other variables that are changing, such as economic, cultural, and institutional variables. In short, they are all likely to be moving in the same direction, either up or down.  For (2), again basically for the same reason, it will prove difficult to isolate the effect of democracy because states with high levels of democracy are also likely to have high levels of economic development, relatively efficient institutions, etc.

I have found this to be a bit disappointing yet very interesting.  As I embark on my PhD in political science in the fall and having come from an economics background, my work in political science tends to be empirically oriented.  Now democracy tends to be the most obvious variable to include in any regression, but if it is relatively stable, does this make any sense?  I know a bit of econometrics, but not enough to qualify as an expert.  So how are stable variables accounted for?  Can we ever really say that a stable variable explains a non-stable variable (or vice versa)?  I don’t think a stable variable can every really explain a non-stable variable, but I really don’t know.

Silent Trade, Game Theory, and The Survivorship Bias

Just read about ‘silent trade’ for the first time.  Its pretty strange.  From what I understand, silent trade occurs when someone (or a group of people) leaves their wares in a neutral position (such as a sea-shore) and a second group collects these goods, whereby they in turn leave their wares for the first group to subsequently collect.  The two groups never come into direct contact, hence the name ‘silent.’

The author of the post is skeptical that silent trade was more than just a myth.  A few game theoretic set ups are used to see if a silent trade equilibrium could be reached and given the inherent flexibility of the game theoretic method, examples of strategies are found that could support either side of the argument.

But i think the explanation for whether or not silent trade existed is easier and using game theory here is just a distraction.  All one needs is the survivorship bias.  For every attempt at silent trade that succeeded, dozens probably failed.  The many more cases of failure would correspond to the defection outcomes in any prisoners dilemma game.  In such cases, no myths would be generated.  But given enough attempts, it seems plausible that (perhaps for one generation) the stars would align and coordination takes place.  Seeing the positive signs from the gods, the traders create their myth and pass it down through the generations until someone writes it down.

In short, there are many individual events that occur (or at least seem to) despite their very low probability.  But the space of all potential low probability events is extremely large, so it would be strange if at least a few of these low probability events did not occur from time to time.  And the survivorship bias will ensure we continue to hear about them.