Paul Romer on mathiness

Top growth theorist Paul Romer has an essay in the AER Papers & Proceedings, in which he comes down harshly "mathiness" in growth theory. "Mathiness" is his term for when people (allegedly) use math in a sloppy way, to support their preferred theories. Romer warns direly that the culture of econ theory has become a lot more tolerant of mathiness:

If mathiness were used infrequently,...it would do localized, temporary damage. Unfortunately...as the quantity increases, mathiness could do permanent damage because it takes costly effort to distinguish mathiness from mathematical theory. 

The market for mathematical theory can survive a few...articles filled with mathiness. Readers will put a small discount on any article with mathematical symbols, but will still find it worth their while to work through and verify that the formal arguments are correct, that the connection between the symbols and the words is tight, and that the theoretical concepts have implications for measurement and observation. But after readers have been disappointed too often by mathiness that wastes their time, they will stop taking seriously any paper that contains mathematical symbols. In response, authors will stop doing the hard work that it takes to supply real mathematical theory. If no one is putting in the work to distinguish between mathiness and mathematical theory, why not cut a few corners and take advantage of the slippage that mathiness allows? The market for mathematical theory will collapse. Only mathiness will be left. It will be worth little, but cheap to produce, so it might survive as entertainment. 

[I]n the new equilibrium: empirical work is science; theory is entertainment. Presenting a model is like doing a card trick. Everybody knows that there will be some sleight of hand. There is no intent to deceive because no one takes it seriously. Perhaps our norms will soon be like those in professional magic; it will be impolite, perhaps even an ethical breach, to reveal how someone’s trick works. 

When I learned mathematical economics, a different equilibrium prevailed. Not universally, but much more so than today, when economic theorists used math to explore abstractions, it was a point of pride to do so with clarity, precision, and rigor...If we have already reached the lemons market equilibrium where only mathiness is on offer, future generations of economists will suffer.

Romer has now joined the chorus of old famous guys - Krugman, Solow, Stiglitz, Farmer - who are very vocally mad about the way mainstream economics theory is done.

Romer is not afraid to name names. Interestingly, although he's talking only about growth theory and not about business cycle theory, most of the people he's mad at are the same guys that the Keynesians are mad at - Robert Lucas, Ed Prescott, and David K. Levine. He also calls out Thomas Piketty.

Romer gives specific examples of what he calls mathiness (links are to working-paper versions):

1. Prescott and McGrattan (2010): Romer says that this paper includes a term that the authors label "location," but that doesn't correspond to any real measure of location.

2. Boldrin and Levine (2008): Romer criticizes this paper for assuming that a monopolist would also be a price-taker, and for making various hand-wavey arguments.

3. Lucas (2009): Romer criticizes this paper for making a hand-wavey argument to dismiss the idea that investment in embodied technology (books, blueprints, etc.) can be a source of sustained growth, when there are well-known models in which it can. Romer also points out a random math error in the paper, and uses this to argue that reviewers don't pay close attention to math.

4. Lucas and Moll (2014): Romer criticizes this paper especially harshly. Lucas and Moll claim that their model, in which there is no creation of new knowledge, is "observationally equivalent" to models in which new knowledge arrives very slowly. Romer shows that the truth of this claim depends on which order you use when taking a double limit. He reveals that he told the authors about the problem, but that they ignored him and left it in the paper.

5. Piketty and Zucman (2014): Romer points out the by now well-known "gross vs. net" problem in Piketty and Zucman's definition of savings.

All in all, this seems like a pretty loose collection of criticisms. Hand-wavey arguments, dubious definitions, bad assumptions, and math errors are all very different things. So this essay at first can seem like a grab-bag of gripes that Romer has with individual rivals' papers.

But I think Romer is on to something about the culture of econ theory, at least in the "macro"-ish realms of growth, business cycle, macro-labor, macro-trade, and macro-tax theory (I don't know nearly as much about the culture of the "micro"-ish fields like game theory, decision theory, I/O, etc.; and I know that finance theory has a very different culture). In these "macro"-ish fields, people seem to view math more as a tool for stylized description of ideas than as a tool for quantitative prediction of observables. 

Romer's examples of "mathiness" are all very recent examples. But going back to earlier models, I don't really see much more tight connection of variables to observables. Yes, in a Solow model you can tie capital K to observable things like structures and machines and vehicles. But you'll be left with a big residual, A. Then you can break A down and extract another term for human capital, H. Can you really measure human capital? Human capital can't be bought and sold on a market; you have to bundle it with other goods. So it's very difficult to get a clean measurement of the value of the existing stock of human capital, the way you could get a clean measurement of the existing stock of delivery trucks. Romer cites human capital as a good example of non-"mathiness", but I don't really see a huge difference between that and the "location" used by Prescott and McGrattan (2010). Maybe a minor difference, but not a huge one. As for the remaining A, there's not really any quantitative way to measure the stock of ideas except as a residual. And as for goofy assumptions, well, any growth model is going to have at least one or two assumptions that would make a newcomer to the econ field throw up her hands in disbelief.

Mathiness isn't anything new, it's just the way these econ fields work. The math is there as a storytelling aid (and possibly as a signal of intellectual ability). I think Karthik Athreya said it best:

My view is that a part of what we do is "organized storytelling, in which we use extremely systematic tools of data analysis and reasoning, sometimes along with more extra-economic means, to persuade others of the usefulness of our assumptions and, hence, of our conclusions...This is perhaps not how one might describe "hard sciences".

Do the guys Romer calls out play a little faster and looser with definitions and rely more on hand-wavey arguments? Oh, I'm sure they do - but that's because they're famous old guys. Writing down hand-wavey stuff is a privilege afforded to famous old guys in every academic discipline I know of. In econ, it gets politely published in top journals, but all the hotshot young people just sort of shake their heads anyway, and the only net effect is to pad out the length of the journals. Are the guys Romer calls out more political than the average economist? Maybe.

But in general, the whole discipline of macro theory - in the general sense, including growth and parts of labor, trade, and tax theory - is chock full of mathiness. Even most of the best models ("best" being a highly relative term, of course). The original Solow model seems to me like a rare exception, not a typical example of the Good Old Stuff.

But in any case, I highly recommend the Romer piece, which is a master class in catching errors in models, as well as a fascinating window into the Byzantine world of academic politics.


Update: Brad DeLong has a follow-up post explaining his view of some of the history behind the argument in the field of growth economics. Basically, the idea is that George Stigler didn't like people using models with imperfect competition, since this might open up a window for government intervention. DeLong thinks that Lucas and other "freshwater" types inherited this anti-imperfect-competition bias, causing them to be too down on Romer's models. This is interesting history that I didn't really know about before.

Update 2: Paul Romer has a response on his blog. Excerpts:

Noah Smith asks for more evidence that the theory in the McGrattan-Prescott paper that I cite is any worse than the theory I compare it to by Robert Solow and Gary Becker... 

There is no such thing as the perfect map. This does not mean that the incoherent scribbling of McGrattan and Prescott are on a par with the coherent, low-resolution Solow map that is so simple that all economists have memorized it. Nor with the Becker map that has become part of the everyday mental model of people inside and outside of economics... 

Noah’s jaded question–Is the theory of McGrattan-Prescott really any worse than the theory of Solow and Becker?–may be indicative of what many economists feel after years of being bullied by bad theory. And as I note in the paper, this resignation may be why empirically minded economists like Piketty and Zucman stay as far away from theory as possible... 

For specific purposes, some maps are better than others. Sometimes a subway map is better than a topographical map. Sometimes it is the other way around. Starting with any good map, we can always increase the resolution and add detail. 

No map is perfect, but this does not mean that all maps are equal. It certainly does not mean that an internally consistent map that with so little detail that you can memorize it is on a par with incoherent scribbling.

In the rest of the post, he goes into depth about why he thinks the McGrattan and Prescott paper constitutes "incoherent scribbling." But he also notes that the other papers he goes after in his "mathiness" piece should not be let off the hook:

Noah also notes that I go into more detail about the problems in the Lucas and Moll (2014) paper. Just to be clear, this is not because it is worse than the papers by McGrattan and Prescott or Boldrin and Levine. Honestly, I’d be hard pressed to say which is the worst. They all display the sloppy mixture of words and symbols that I’m calling mathiness. Each is awful in its own special way.

He also has another short post about a Lucas paper.

It appears that Prescott, Lucas, Levine, and others of the unofficial "freshwater" club have annoyed more high-level colleagues than just Paul Krugman. Only a few months ago, Roger Farmer took to his blog to unleash an anti-Prescott blast. Romer and Farmer are not politically-minded media-engaged types like Krugman and DeLong, but their aggravation with the Lucas/Prescott school is, if anything, even more intense.

And yes, Romer is right that I'm jaded. Is it so obvious? *takes swig from hip flask, rubs beard stubble*

Department of "Huh yourself!": British demand edition

Brad DeLong thinks I'm nutty when I say that Britain isn't obviously suffering from a persistent shortfall of aggregate demand:

There are no signs looking at wages and prices that this is due to any adverse supply shock...there seem to be no reasons looking at wages, prices, and output to believe that the British economy right now is a full-employment at-capacity-utilization economy. And only in such an economy would monetary and fiscal policies that boost spending simply boost prices and not production...

Does the fact that the employment share of British adults is actually high mean that the British economy is, in fact, at potential output? That stimulative monetary and fiscal policies risk rising inflation for no gain? And that it is time to normalize? Certainly Mark Carney at the Bank of England does not believe that is so:

From my point of view, why so many Britons have taken so many low-pay low-productivity jobs in the past three years is a mystery. But that they have gives us little reason to think that the British economy is now a full-employment at-capacity-utilization economy in which aggregate demand is now equal to potential output.

First of all, let's get a couple things straight. I don't think British stimulus would be particularly counterproductive, I just don't think austerity would be either. If there's not a big demand gap, multipliers shouldn't be large, so stimulus just won't do a heck of a lot (unless the UK has falling-apart infrastructure like we do), but neither will austerity. Also I doubt AS-AD is even always the right model for the macroeconomy.

But with that said...

With a demand shortfall, we ought to see high unemployment. We also ought to see low inflation.

In the U.S. we saw both of these in 2009-2014. In Britain we saw the former in 2009-11, and we never really saw the latter at all. Here is British core inflation:

The Bank of England's inflation target is 2% (whether that's a ceiling or a target is not certain). But in 2010-2013 - four years!! - UK core inflation was above that target. 

So if we stick to the good old Econ 102 AS-AD model, and we look at both prices and quantities, we can come up with the following simple story for the UK:

1. In late 2008 the UK suffered a negative AD shock.

2. Around the same time, the UK suffered a negative AS shock.

3. In early 2010 the negative AD shock began to abate.

4. In 2012 the negative supply shock began to abate.

5. The differences between the UK and the U.S. in GDP, employment, and inflation can be explained by the fact that the UK's AD shock wasn't quite as big or long-lasting, while the U.S. didn't experience a supply shock.

This story also fits what we know about British TFP, which declined from 2007-2009, then flatlined through 2011:

This story is incredibly simplified, and uses a model that probably isn't the best. A real, careful, academic analysis of the situation is certainly warranted. But is there some obvious reason we need to go looking for a story where AD is the only thing moving around here? That story is going to have a lot more moving parts, and it's going to go a lot deeper into micro stuff. 

And it will also look like reaching. Why should we demand a story where demand is the whole story? Is this about stabilization policy, or about the size of the British state?

Economists as all-purpose sages: The case of Freakonomics

The original book Freakonomics, by Steve Levitt and Stephen Dubner, was a very fun read. But it also slightly annoyed me. Why? Because there's very little actual economics in it! The quantitative empirical work is mostly reduced-form regressions with natural experiments. That's a fine and good research technique, but it's not really special to econ - it doesn't include anything about market design, structural estimation of supply and demand, game theory, search, prices, general equilibrium...nada!

That covers three of the six chapters. The other three include 1) an ethnography of drug dealer culture by a sociologist, the excellent Sudhir Venkatesh, 2) a quick gloss of statistics techniques that applied mathematicians use to catch cheaters, and 3) a historical story about the decline of the KKK.

So this book has sociology, history, stats, and some general empirical techniques that could be used by any social scientist. That doesn't make it bad - most of the research the book showcases is really cool (though Levitt's own study, on abortion and crime, ended up having some serious problems). But it means that an empirical sociologist could easily taken Levitt's place as the technical co-author of the book, alongside journalist Dubner.

But it was an economist Dubner got, and Freakonomics was billed as a pop econ book, not a pop sociology book. Why? It seems to me that it's because economists are respected as all-purpose sages. Like I said in my previous post, economists get taken seriously on any topic imaginable.

To use an even more stark example, take the sequel, Superfreakonomics. There's a chapter in that book that's all about geoengineering. That's an engineering topic. A physics topic. A climate science topic. And yet an economist is put forth as an authority on whether it will work. And this is accepted by the book's legions of fans.

People trust economists on any topic.

Why? I don't know, to be honest. Maybe it's because economists are thought to have high IQ and know a lot of math relative to other social science disciplines. Maybe it's because economists are confident in their ability to model any social phenomenon, like Gary Becker and others of the "imperialist econ" school. Or maybe it's because economists are just more willing to engage with the public and hold forth on any topic. After all, op-ed writers are the other group who are treated as all-purpose experts; maybe economists just act like really hi-tech, super-smart op-ed writers.

Or maybe it's because of economists' reputation as being clear-eyed and impartial observers of society. For example, in the intro to Freakonomics, Dubner describes a scene with Steve Levitt:

An elderly homeless man approaches [Levitt's car]. It says he is homeless right on his sign, which also asks for money. He wears a torn jacket, too heavy for the warm day, and a grimy red baseball cap. 

[Levitt] doesn’t lock his doors or inch the car forward. Nor does he go scrounging for spare change. He just watches, as if through one-way glass. After a while, the homeless man moves along. 

“He had nice headphones,” says [Levitt], still watching in the rearview mirror. “Well, nicer than the ones I have. Otherwise, it doesn’t look like he has many assets.”

If you didn't know Levitt was an economist, this scene would just make him sound like a rich insensitive jerk. But the fact that he's an economist imbues the scene with a different meaning altogether - suddenly, Levitt's clinical detachment seems like a sign of impartiality and rationality. Exactly the qualities we'd want in an all-purpose sage.

Anyway, I don't know the answer. But the observation that people trust economists on any social topic - and even some engineering topics - seems pretty obvious. And pretty freaky.


P.S. - If you want a book that is nothing but hardcore economics, and explains everyday economic phenomena in a way that is humble, entertaining, and useful all at the same time, I recommend Tim Harford's The Undercover Economist. The best version is the audio version.