Should theories be testable?

Lots of interesting philosophy-of-science arguments around the web these days.

In physics, there's an editorial in Nature complaining about string theory - and especially the "string theory landscape" - isn't falsifiable. (Personally I think the word "falsifiability" is a little silly, since it's just testability + strongish priors against your own hypothesis.) Brian Greene, a string theorist, has a response in Smithsonian magazine. On Twitter, Sean Carroll complains about the "falsifiability police". Personally, I think Chad Orzel has the best take on the whole thing. 

I don't see why we should insist that any theory be testable. After all, most of the things people are doing in math departments aren't testable, and no one complains about those, do they? I don't see why it should matter if people are doing math in a math department, a physics department, or an econ department. 

I think testability starts to matter when you start thinking about applying theories to the real world. This is why I get annoyed when people ignore the evidence in business cycle theory, but not when they do it in pure theory. 

Suppose you're studying the properties of repeated games. Who cares if those games represent anything that really exists today? They might represent something we might implement with algorithms somewhere in the future. Or even if not, it's fun (i.e. valuable) to just know a bunch of cool stuff about how concepts fit together (i.e. math). The same is true about the kind of abstract "math of value functions" stuff that Miles Kimball taught me in grad school.

But when you start making models that claim to be about some specific real thing (e.g. monetary policy), you're implying that you think those models should be applied. And then, it seems important to me to have some connection to real data, to tell if the theory is a good one to use, or a crappy one to use. That's testability.

Anyway, this sort of seems very college-freshman-dorm-discussion-level when I write it out like this, but I think there are a surprising number of people who don't seem to agree with it...

Elsewhere, Kevin Bryan has a post up about "minimal model explanations" in economics, which basically echoes Friedman's "methodology of positive economics". Brad DeLong links to an Itzhak Gilboa paper about economic models as analogies. Moises Macias Bustos informs me that the Stanford Encyclopedia of Philosophy has updated its entry on "scientific explanation". And Robert Waldmann reminds me of this interesting post, in which he argues that Friedman's ideas and Lucas' ideas about economic methodology are mutually contradictory.


Update: I think it's worth pointing out once again that purely mathematical theories, which don't describe any pre-existing phenomenon (and hence are not "testable"), can be useful.

A good example is the Stable Matching Theory developed by Al Roth and Lloyd Shapley. When this theory was developed, it didn't describe anything that existed in the world. So you couldn't go out and test it. It was obvious that it "worked", in the sense that you could program computers that implemented it. That was trivial. You could know that just from working out the math. So this theory, when it was made, wasn't a "testable" theory like General Relativity. But then, eventually, people came up with a way to use Stable Matching Theory for assigning organ transplants. And it worked really well. So it turned out to be useful.

Now look at a lot of the stuff people are doing in math departments. How much of that stuff will eventually be useful? The answer is "We don't know, and we can't know." In 1896, in a letter discussing the new theory of vectors, Lord Kelvin - one of history's greatest physicists - said "'[V]ector' is a useless survival...and has never been of the slightest use to any creature." To put it mildly, he was extremely wrong.

So abstract, mathematical "theories" that can't be tested like science theories can still be useful. And we can't know which of them will be useful in the future. And it's cheap and harmless to have people sit around and work on those things. And I can't see how it matters whether those people are in math departments, or physics departments, or econ departments, or computer science departments, or statistics departments, or applied math departments, etc.

But as soon as people start saying - or even implying - that their theories describe real phenomena, then the ball game changes.