Image from Inception/Be Inspired Channel

By Professor Glauco De Vita, Centre for Business in Society

‘Replicability’ is an important part of every science, though what is meant by replication varies quite markedly across disciplines and even within the economics field is still the subject of debate (see, e.g., Duvendack et al., 2015; Clemens, in press). As stated by McCullough and Vinod (2003: 888) over a decade ago: “Research that cannot be replicated is not science, and cannot be trusted either as part of the profession’s accumulated body of knowledge or as a basis for policy”. Nevertheless, with few notable exceptions (e.g., De Vita and Trachanas, 2016; Herndon et al., 2014) verification through replication remains, disappointingly, an undervalued endeavour of economics research. Indeed, aside from sporadic ‘Comments’ or ‘Notes’, standard, full-length replication articles rarely appear in the pages of top journals.

In their strong call for more replications, Burman et al. (2010: 788) emphasise that replication is a critical tool for scientific progress and that the absence of such studies “is particularly problematic because empirical economic research is often prone to error.” Clearly, there is a need for investing in transparency and reproducibility. First, by authors themselves, who should provide the data and any further material necessary to replicate or re-analyse the published results. Second, by the editors of top journals, who should tighten up their data archives policy and place due consideration to ‘Comments’ or rigorous replications of papers published in their journals.

The Journal of Human Resources, Journal of Political Economy and the American Economic Review were the ones that led the way on this front. Although over recent years the economics profession has made further progress, journals prepared to openly put under public scrutiny the reliability of findings published in their pages, remain a minority. There are exceptions, of course. For example, the above-mentioned article by De Vita and Trachanas published by Energy Economics, or the Journal of Applied Econometrics, which now has a special section specifically devoted to publishing successful as well as unsuccessful replications and re-analyses. A special mention goes to Econ Journal Watch. Founded in 2003, its main remit is “to watch the journals for inappropriate assumptions, phony claims of relevance, omissions of pertinent truths, and irreplicability” (https://econjwatch.org/about/). Arguably, transparency and reproducibility are needed more than ever before as they are a conditio sine qua non to help enhance the credibility we need in economics in order to make use of the knowledge we create.

Doubtless, the call for more replications is not one to be embraced without caution in social science. First, there is a need to homogenise the terminology to ensure a common understanding of the meaning of replications. As recently observed by Clemens (in press), “A discrepant replication, in current usage of the term, can signal anything from an unremarkable disagreement over methods to scientific incompetence or misconduct.” Second, the value commonly attributed to replications, and hence the way we think about them, should change. Far from being a basic task to be delegated to doctoral students to help them refine their econometric skills with specific estimation packages, ‘work of replication’ should be rightly thought of as the ultimate challenge of applied economists at the top of their profession. As it had been most accurately described to me over twenty years ago by a Cambridge professor, publishing a replication study in a top journal is “the most challenging task in economics research given the standards of rigour required to convincingly disprove, beyond reasonable doubt, previous evidence published in that journal.”

Yet incentives remain low to undertake replications, however valuable they are in the incessant search for truth and knowledge. Replications are still perceived by too many stakeholders within the academic milieu as ill-fitting “within the ‘puzzle-solving’ paradigm which defines the reward structure in scientific research” (Dewald et al., 1986: 587). On this account, it should be recognised that the academic worth of the finest replications goes well beyond invalidating previous results. It entails the nullification of influential yet misleading policy implications which carry non-trivial consequences for any policy guidelines of widespread significance for the economy and society at large. The value of the contribution of a replication, therefore, is at least on a par with that attributed to the original article subjected to scrutiny.

References

Burman LE, Reed WR, Alm J, 2010. A call for replication studies. Public Finance Review 38, 787–793.

Clemens MA, in press. The meaning of failed replications: A review and proposal. Journal of Economic Surveys. doi: 10.1111/joes.12139.

De Vita G, Trachanas E, 2016. ‘Nonlinear causality between crude oil price and exchange rate: A comparative study of China and India’ – A failed replication (negative Type 1 and Type 2). Energy Economics 56, 150–160.

Dewald WG, Thursby JG, Anderson RG, 1986. Replication in Empirical Economics: the Journal of Money, Credit and Banking Project. American Economic Review 76, 587–603.

Duvendack M, Palmer-Jones RW, Reed WR, 2015. Replications in Economics: A Progress Report. Econ. Journal Watch 12, 164–191.

Herndon T, Ash M, Pollin R, 2014. Does high public debt consistently stifle economic growth? A critique of Reinhart and Rogoff. Cambridge Journal of Economics 38, 257–279.

McCullough BD, Vinod HD, 2003. Verifying the solution from a nonlinear solver: A case study. American Economic Review  93, 873–892.

Comments

comments