Influential Reinhart-Rogoff economics paper suffers spreadsheet error

Influential Reinhart-Rogoff economics paper suffers spreadsheet error

April showers bring … database errors?

The other day, we wrote about two retractions in the Journal of the American College of Cardiology, and another in the American Heart Journal, stemming from database errors.
Seems to be catching.
The Economist (among other outlets) this week is reporting about a similar nother database glitch — not, we’ll admit, a retraction — involving a landmark 2010 paper by a pair of highly influential economists. The controversial article, “Growth in a Time of Debt,” by Harvard scholars Carmen Reinhart and Kenneth Rogoff, argued that countries that took on debt in excess of 90% of their gross domestic product suffered sharp drops in economic growth. That evidence became grist for the austerity mill, including Paul Ryan.
Turns out, that conclusion was based to some extent on an Excel error. As the business press explained, a trio of researchers at the University of Massachusetts,  Thomas Herndon, Michael Ash and Robert Pollin, found that the Reinhart/Rogoff anlysis had excluded a handful of critical data points by basically lopping them off the spreadsheet. The result: Their claims about the deleterious effects of debt on growth are somewhat — indeed, substantially, in a way — overstated.
As the U. Mass economists note in their rebuttal paper, “Does High Public Debt Consistently Stifle Economic Growth? A Critique of Reinhart and Rogo ff“:

Herndon, Ash and Pollin replicate Reinhart and Rogoff and find that coding errors, selective exclusion of available data, and unconventional weighting of summary statistics lead to serious errors that inaccurately represent the relationship between public debt and GDP growth among 20 advanced economies in the post-war period. They find that when properly calculated, the average real GDP growth rate for countries carrying a public-debt-to-GDP ratio of over 90 percent is actually 2.2 percent, not -0:1 percent as published in Reinhart and Rogo ff. That is, contrary to RR, average GDP growth at public debt/GDP ratios over 90 percent is not dramatically different than when debt/GDP ratios are lower.
The authors also show how the relationship between public debt and GDP growth varies significantly by time period and country. Overall, the evidence we review contradicts Reinhart and Rogoff ’s claim to have identified an important stylized fact, that public debt loads greater than 90 percent of GDP consistently reduce GDP growth.

In the interest of full disclosure, we’ll also quote the two corrections to this paper:

(1) The notes to Table 3: “Spreadsheet refers to the spreadsheet error that excluded Australia, Austria, Canada, and Denmark from the analysis.” is corrected to read: “Spreadsheet refers to the spreadsheet error that excluded Australia, Austria, Belgium, Canada, and Denmark from the analysis.”
(2) Page 13: “Thus, in the highest, above-90-percent public debt/GDP, GDP growth of 4.1 percent per year in the 1950-2009 sample declines to only 2.5 percent per year in the 1980-2009 sample” is corrected to read “Thus, in the lowest, 0–30-percent public debt/GDP, GDP growth of 4.1 percent per year in the 1950–2009 sample declines to only 2.5 percent per year in the 1980–2009 sample.”

Reinhart and Rogoff, for their part, have acknowledged the error:

We literally just received this draft comment, and will review it in due course. On a cursory look, it seems that that Herndon Ash and Pollen also find lower growth when debt is over 90% (they find 0-30 debt/GDP , 4.2% growth; 30-60, 3.1 %; 60-90, 3.2%,; 90-120, 2.4% and over 120, 1.6%). These results are, in fact, of a similar order of magnitude to the detailed country by country results we present in table 1 of the AER paper, and to the median results in Figure 2. And they are similar to estimates in much of the large and growing literature, including our own attached August 2012 Journal of Economic Perspectives paper (joint with Vincent Reinhart) . However, these strong similarities are not what these authors choose to emphasize.
The 2012 JEP paper largely anticipates and addresses any concerns about aggregation (the main bone of contention here), The JEP paper not only provides individual country averages (as we already featured in Table 1 of the 2010 AER paper) but it goes further and provide episode by episode averages. Not surprisingly, the results are broadly similar to our original 2010 AER table 1 averages and to the median results that also figure prominently.. It is hard to see how one can interpret these tables and individual country results as showing that public debt overhang over 90% is clearly benign.
The JEP paper with Vincent Reinhart looks at all public debt overhang episodes for advanced countries in our database, dating back to 1800. The overall average result shows that public debt overhang episodes (over 90% GDP for five years or more) are associated with 1.2% lower growth as compared to growth when debt is under 90%. (We also include in our tables the small number of shorter episodes.) Note that because the historical public debt overhang episodes last an average of over 20 years, the cumulative effects of small growth differences are potentially quite large. It is utterly misleading to speak of a 1% growth differential that lasts 10-25 years as small.
By the way, we are very careful in all our papers to speak of “association” and not “causality” since of course our 2009 book THIS TIME IS DIFFERENT showed that debt explodes in the immediate aftermath of financial crises. This is why we restrict attention to longer debt overhang periods in the JEP paper, though as noted there are only a very limited number of short ones. Moreover, we have generally emphasized the 1% differential median result in all our discussions and subsequent writing, precisely to be understated and cautious, and also in recognition of the results in our core Table 1 (AER paper).
Lastly, our 2012 JEP paper cites papers from the BIS, IMF and OECD (among others) which virtually all find very similar conclusions to original findings, albeit with slight differences in threshold, and many nuances of alternative interpretation.. These later papers, by the way, use a variety of methodologies for dealing with non-linearity and also for trying to determine causation. Of course much further research is needed as the data we developed and is being used in these studies is new. Nevertheless, the weight of the evidence to date –including this latest comment — seems entirely consistent with our original interpretation of the data in our 2010 AER paper.
Carmen Reinhart and Kenneth Rogoff
April 16, 2013

1945-2009
	RR (2010)		HAP (2013)
Debt/GDP	Mean	Median	Mean	Median
0 to 30	4.1	4.2	4.2	NA
30 to 60	2.8	3.9	3.1	NA
60 to 90	2.8	2.9	3.2	NA
Above 90	-0.1	1.6	2.2	NA
1800-2011
1800-2011
0 to 30	3.7	3.9	NA	NA
30 to 60	3.0	3.1	NA	NA
60 to 90	3.4	2.8	NA	NA
Above 90	1.7	1.9	NA	NA
RRR (2012), 1800-2011
	Mean
Below 90	3.5
Above 90	2.4

Of course, this isn’t a retraction, at least not yet. And as we’ve noted, retractions are rare in economics. Still, a number of people have sent us tips about this paper, so we thought it was worth a post.
Update, 10:15 a.m. Eastern, 4/19/13: Word in headline corrected from “database” to “spreadsheet,” as per comment below from Neil Saunders.

Retraction Watch
2013.4.18

写し

--

経済学でもこんなのがあります。