Journals, editors, referees, and authors: experiences at the Journal of Economic Literature.

Pencavel, John

I have been invited to write about my experiences as editor of the Journal of Economic Literature (JEL) and to reflect on what these experiences may mean for the status of journals in intellectual inquiry. I was editor of the JEL for thirteen years from 1986 to 1998. Before becoming editor, for four years, I served as Associate Editor under the editorial supervision of Moses Abramovitz. After stepping down as Editor, I was a member of the Board of Editors of the JEL until 2006. Therefore, I was associated with the administration of the Journal for almost 25 years--from 1982 to 2006.

What I write here focuses on my years as editor. However, as Associate Editor, I learned a great deal from Moe Abramovitz and I am indebted to him for the model he provided me of a conscientious and active journal editor. In effect, my years as an Associate Editor were something of an apprenticeship to an eminent and honorable scholar and I benefitted greatly from the training and education I received from Moe.

The JEL Survey Articles

First and foremost, I must write that the Editorship of the JEL was largely a labor of love. No doubt, I worked hard at the position. I took a deep personal interest in the well-being of the journal and invested much effort in it. However, this was not selfless. There were substantial private returns. Although my Department at Stanford had kindly assigned me to teach a graduate microeconomic theory class, I felt my knowledge of Economics was becoming increasingly specialised. I knew more and more about a narrower scope of Economics. The editorship of the JEL represented an opportunity to counter this professional imperative towards specialisation. As editor of JEL, I invited articles on topics that I knew little about and that I wanted to become much better informed of. Of course, this required a prior investment in reading about these topics so I could identify the appropriate people to approach and perhaps to write these articles. In this, my Board of Editors was often very helpful in proposing particular people or in guiding me away from unsuitable writers. The Board consisted of economists whom I had selected and whom the AENs Executive Committee approved. The members of the Board had different specialties and they shared with me the goal of publishing articles that informed non-specialist economists about important research in particular fields of Economics.

I would approach a potential author and declare my interest in an accessible survey paper directed to non-specialist economists. I encouraged the likely author to sketch an outline of the paper he or she would deliver. I asked that the paper not be organized around names and particular papers as if it were a string of abstracts pieced together such as "X (1980) claimed this, Y (1984) argued that, and Z(1989) responded in this way". This is tedious to read, if not to write. I asked for a selective and synthetic review of a major research effort in which a necessary ingredient would be the evaluation of this research endeavour: what have been the successes and the failures in this line of research?; how much confidence can be placed in the literature's findings?; what do we think we know and what do we believe we do not know?; and where should future research efforts be directed? This element of evaluation ought to be of value to the specialists. Here is an opportunity to step back from the research frontier and to take stock of a significant intellectual enterprise and to pass judgment on it. In this way, a successful JEL article would speak both to the specialist and to the non-specialist economist.

Because the paper was designed for the non-specialist, if it were appropriate, I would not hesitate to scrawl across the page of a submitted draft "I don't follow this". For a paper in Labor Economics, my area of expertise, I would be more likely to scrawl, "This won't be understood by the non-specialist". No doubt, this insistence on a paper that was accessible to the non-specialist and, as a consequence, the many drafts that some papers went through to satisfy my requests for expository clarity did not endear me to certain authors, some of whom gave up and took their work product elsewhere. However, I believe that, in most cases and for those papers that were ultimately published in the JEL, this concern with expository clarity enhanced the paper's readership and influence. I have always believed that exposition matters, even in the most technical of articles, and it is surely no coincidence that the discipline's unquestioned intellectual giants of the past fifty years or so have been very effective expositors.

One quality lauded in the academic community is the ability to analyse arguments and propositions in a detached and dispassionate manner. However, it is an unusual author of a manuscript who shows the necessary detachment and disinterest in his own paper. I have found that, at the stage of floating arguments and testing chains of thought in discussion, scholars are often quite disinterested and detached. But once the scholar puts down these arguments and claims in writing, often he becomes attached to his papers almost as are mothers to their babies. It is as if the act of expressing these arguments and claims in written word elevates the papers to a position calling for deference and respect. For the editor, this means that suggestions for changes in papers are best couched to authors gently and respectfully, perhaps indirectly. The most effective way to induce changes in a submitted manuscript was to treat all papers with deep respect and high regard even if this sometimes required me to shed my personal beliefs and prejudices and adopt a more neutral stand.

No doubt, under my Editorship, some weak articles were published and some very good articles were omitted. These are the errors that any editor makes. I think that these two types of errors do not operate on editors with equal force. The error of publishing an inadequate or poor article is much more visible and conspicuous than the error of failing to publish an excellent article. Hence editors (and even the referees) tend to "play it safe" and they are inclined to turn away potentially controversial papers or those that receive mixed reviews or those that are out of the mainstream. Surely this asymmetry in potential losses helps to account for the famous journal rejections of what, in retrospect, turned out to be seminal pieces of work. (2) I was aware of this asymmetry and tried to lean against it, but nevertheless it would be remarkable if I never turned away what would have been excellent publications.

After the Journal of Economic Perspectives (JEP) was established, I was often asked how the ideal JEL article differed from that of its sister journal, the JEP. I thought of the prototypical JEL article as a thorough review of a class of research and one that articulated different points of view. The author of this article was encouraged to come down on one side or the other of a controversy, but this judgment should be expressed after a fair and balanced characterization of other points of view. By contrast, I saw the prototype JEP article as one in which the author was encouraged to advance his or her perspective on a research venture. Balance was achieved in the JEP by having several articles by various authors on a given topic, not by one article describing and assessing different judgments. Indeed, several JEP editors expressed this same goal for their articles.

The Referees

The process of scientific inquiry is, above all, a social activity. Each of us may sit isolated for years developing an argument. But, ultimately, to constitute a recognized contribution to knowledge, the argument has to meet a social test: to be evaluated and critically examined by others in the intellectual community. It is here where the journals perform an essential function of review and information dissemination. (3) The JEL's role is not so much that of publishing articles containing original frontier research but that of communicating an important class of scholarship to a wider intellectual community and of providing dispassionate assessment of that scholarship. The JEL stands as an attempt to keep our horizons broad and, in this way, strengthen the bonds of the Economics community by sharing knowledge among us.

I owe a deep debt to many referees I called upon for advice. A number of referees wrote remarkably thorough and discerning reports. Their anonymous contributions to many published papers were often substantial. My practice was to invite the referees into the paperwriting process early. I would send referees the outlines of proposed papers and seek their assessments of the projected article. When suitable drafts of the paper were delivered, I went back to the same referees for their assistance which was usually provided very generously. My practice was to speak to each referee on the telephone before sending the draft to him or her. At that time, I would ask for a delivery date for his report and would telephone him if his report was not delivered by his chosen date. In this way, referee reports were garnered in a few months and no author ever waited more than twelve weeks for a referee-evaluated response to his submitted JEL paper. Most authors heard from me within two months of the submission of their papers.

This turnaround time may have been shorter than many other journals but not significantly so for a leading journal in the 1980s. However, this turnaround time seems to have become increasingly rare and editorial practices are far less conscientious today. In part this is because of the increase in volume of manuscripts that these journals now handle, but this helps explain why referees may be more tardy; it doesn't account for why editors are willing to allow their authors to hang on in ignorance of what is going on.

I myself submitted a paper to a respectable journal within the past two years. I received no word from the editorial office that the paper had arrived nor any word of the disposition of my paper. Exactly one year after submitting the paper, I wrote to the journal seeking confirmation that the paper had been received and asking what had happened to my submission. The identity of the co-editor who was handling my manuscript was revealed to me but, when I contacted him, he simply ignored my request for information. Then, after sixteen months, I received two referee reports (until that time, I did not know if my paper had been sent to referees) and an apology from the co-editor for the referees' tardiness. Of course, I had not been writing about the referees' tardiness; I was complaining about the co-editor who lacked the professional courtesy to inform me of the status of my paper.

Colleagues have described some comparable, less extreme, cases with other journals which leads me to conjecture that, while such occurrences were unusual twenty years ago, they are commonplace today. Why? I suspect one important reason is that editors today are less inclined to see their professional standing wrapped up in the success or failure of the journal they edit. Nowadays, a journal editorship is one job among many and inefficient practices and discourteous treatment of the authors of submitted papers constitute the necessary by-product of the modern scholar juggling many different activities. Certainly the number of journal editors and co-editors seems to have increased though I do not know whether the ratio of the number of journal editors and co-editors to submitted papers has increased. Twenty years ago, the editorship of a major journal in Economics was an honor, but today my conversations with some journal editors lead me to wonder whether the honorific quality of the job has tarnished. In many cases, an editorship appears to be viewed primarily as a burden and, under these circumstances, it seems an author must tolerate what little courtesy that an editor may allot to him.

I have often said that my experience as editor convinced me that "the referee is always right". This is a remarkable claim and, of course, there are lots of examples of this statement being false. What I mean by this somewhat extreme aphorism is that, if a reputable referee has spent a considerable amount of time and effort in understanding a paper and if the referee asserts that the paper is arguing one thing whereas the author insists he has written something else, then the author has failed to persuade a sceptical yet serious colleague--the referee--of the soundness of the argument and this calls for a better and more effective exposition by the author. In other words, it is not a bad rule of thumb for an editor to act as if the referee's claims are correct and to ask the author of the paper to explain his argument more carefully so that a diligent referee should not draw the wrong inference.

The JEL's Bibliographic Activities

Of course, the JEL is known not only for its authoritative survey articles but also for its bibliographic activities such as book reviews, periodical abstracts, and the contents of major journals. Indeed, some of my most important decisions as editor involved keeping EconLit, the bibliographic reference source, within the ownership of a professional association, the American Economic Association, rather than selling the rights to a for-profit company such as Elsevier. Who knows what the prices of EconLit would be today if it were not owned and managed by the AEA but owned and managed by Elsevier?

The information contained in EconLit was ideal for distribution through the Internet and, indeed, this became the source of considerable revenues for the American Economic Association. Our first step was to license CD-Roms of EconLit by Silver Platter and these licensing agreements not only distributed a useful resource to economists but also generated substantial income for the AEA. (4) The subsequent development of on-line access to EconLit has provided more largesse for the Association. The revenues generated from EconLit have been used to keep the prices of the flagship journals relatively low and to embark on the production of both the Journal of Economic Perspectives and, more recently, of the AEA-sponsored field-specific journals. When I encouraged the development of EconLit on CD-Rom, I did not realize how fruitful for the Association it would become. Both because EconLit is a monopoly of information about Economics literature and it has public goods features, I always had the firm belief that it should remain the property of the notfor-profit AEA and not of a for-profit commercial venture. (5)

One aspect of the bibliographic arm of JEL concerned the classification by subject-matter of books and articles in Economics. When I became editor, the classification system was several decades old and I felt it was not well suited to contemporary Economics. I embarked on designing a new classification system. Initially I proposed an overarching classification distinguishing the major fields within Economics such as International Economics, Public Economics, and Economic History. Then, within each of these major fields, I enlisted the assistance of specialists to propose a finer classification system and I distributed widely the various proposals I received. Many people were involved and I received very constructive contributions. Perhaps the involvement of so many scholars explains why the new system (launched in 1992) took over so quickly and why, except for modifications here and there to accommodate the shifting boundaries and expansion of the discipline, the basic structure remains in effect today. It is used by other Economics associations as the basis for their own classifications and, more often than not, the classification system introduced in 1992 finds wide international application.

The Contents of Current Periodicals

When I was editor and before the widespread use of the internet to convey large quantities of information, the hard copy of the JEL contained sections titled "The Contents of Current Periodicals" and "Abstracts of Selected Articles". Nowadays this appears only on-line and not in the hard copy version. These sections of the JEL contained the major journals of Economics (organized alphabetically) and listed the articles that were published in each recent issue. In the early 1990s, we listed the contents of about 330 journals and, for a subset of these journals, we also published abstracts of the published papers. My job involved, in part, the responsibility of determining those periodicals selected for listing and, from these listed periodicals, those selected for abstracting. This task of selection probably induced more outrage from certain journal editors than anything else I did as JEL editor. This is because each journal sought to be included in the list of major journals; a listing in the JEL became the mark of approval that the Journal of Economic Nonsense had attained the ranks of the premier periodicals in Economics. This was a period in which the number of journals was growing and I was often receiving requests from editors for their journals to be listed in The Contents of Current Periodicals. In earlier days before I became editor, JEL Board of Editor meetings were taken up with long discussions over the merits of various journals--or so I was told. I felt I needed some mechanical procedure or dispassionate rule to determine which journals ought to have their contents listed and which ought not. The criterion I developed was a measure of the citations that a journal's articles were receiving from other journals' articles.

When I became editor, I initiated a thorough review of our procedures both for listing and for abstracting. It was clear that some journals were included in our lists simply because they had qualified for inclusion back in the 1960s when the Journal of Economic Literature started publication and there had not been a serious reevaluation of their contents. In other cases, I wondered whether a journal's inclusion reflected some favor to a friend or the personal predilections of members of the Journal of Economic Literature's Board of Editors. The reevaluation of the journals involved computing many different citation indicators to measure the impact of each and every journal. (6) This was completed by the early 1990s and it led to changes in the JEL's procedures. This work led me to address the sort of questions characteristic of industrial organization: viewing each journal as if it were a firm's product and defining the industry as the set of all such firms (journals), we may ask:

1. is the industry competitive?;

2. is the industry concentrated in a few firms?;

3. are there barriers to entry?;

4. how profitable are the firms?; and

5. has innovation affected the structure of the industry?

Competition?

The opportunities for scholars to publish and the opportunities for economists to read about research are tremendous. Not merely is every specialty and style represented in these journals, but even within specialties there are many different outlets for research. If each journal is a product type, then economics journals satisfy the first requirement of a highly competitive industry: there are very many products. These products are not the same, but there are sufficiently close substitutes in the market for the industry to be described as highly competitive.

Concentration?

At the same time, along some dimensions, the market for economics articles can be said to be highly concentrated. In what sense is this true? In my analysis of various citation indices, a relatively small set of journals provide most of the citations. Indeed, many journals publish articles that are barely cited at all. Many journals have issues in which none of the published articles is subsequently ever cited by articles published in other journals!

At the other end of the spectrum, using one such citation index, the top ten percent of journals accounted for 88% of all citations and the top nine journals accounted for half of all citations! In other words, a very large fraction of all citations are concentrated in a small number of journals and it is in this respect that the market for economics journals may be said to be concentrated. Note that, in this sense, concentration is fully compatible with competition: though most citations are to articles in a small number of journals, there is keen competition among these quality journals for articles that are potential candidates for publication in a number of them.

Barriers to Entry?

If citations are concentrated in a few quality journals, has the composition of the quality journals changed over time? I do not know the answer to this as the citation analysis I undertook was restricted to the 1980s. However, let me offer a conjecture: there has been a certain asymmetry such that it has been easier for journals to enter the class of high quality journals than for journals to fall out of that class. Most would classify Econometrica and the Journal of Political Economy as top tier journals today and, indeed, they would have been so judged fifty years ago also. However, other journals that would be judged today as quality journals were not in operation fifty years ago including the Journal of Economic Perspectives and the Journal of Economic Literature. As the profession has grown in size so more quality journals can be supported. Hence, while it might be argued that no journal classified as a quality journal fifty years ago has fallen from that status, other journals have joined that exclusive club.

Of course, some journals are recognized as being important producers of information within particular specialties. Indeed, there are obviously few barriers to the mere publication of journals--I wish there were more!--and it has not been difficult for some journals to establish a very high reputation within specialties. This has surely been the big change over fifty years: the growth in the number of specialty journals which testifies to Adam Smith's maxim that specialization increases with the size of the market.

However, with respect to the top quality, general interest, journals, it is noteworthy that none of those that would be classified as top quality fifty years ago has dropped out of that class. This development might be portrayed as the extension of the aristocracy: the class system of journals is characterized by downward immobility out of the aristocracy, but upward mobility into the aristocracy!

Advantages of Incumbency?

This downward immobility out of the aristocracy suggests to me that there are genuine advantages attached to incumbency. No doubt, a reckless editor who holds views that are unrepresentative of the profession and that he imposes on the articles he publishes has the power to ruin the aristocratic status of one of the quality journals. However, short of this, once an aristocratic journal has attained that state, it appears to require of the editor no more than to behave as an arbitrator between authors of submitted papers and referees in order that the quality of the journal be preserved.

I hasten to add that a good editor does act as more than a plain arbitrator. However, the fact that mere arbitration is sufficient to preserve the status of an aristocratic journal indicates there are considerable advantages to incumbency: once the journal has attained that aristocratic status, it is largely sheltered from challenges that will cause its demotion. The journal's status earned over many years of publication is something that cannot be captured by a new entrant. Such advantages constitute a rent.

What are the forms of these rents and who enjoys them? The rents may consist in part of monetary returns--salaries to the editors and staff, payments to referees or authors--and the fact that profit-seeking publishing companies seem eager to introduce new journals suggest that they see monetary returns in the activity. But, at least among the high quality journals, the rents are not entirely monetary. Some of these rents are non-monetary and, in particular, the satisfaction of being involved in the distribution of information that is highly regarded by peers within your profession.

Innovation?

Finally, let us note how, in many activities over the past century or so, a radical innovation (either in production technologies or in types of products manufactured and sold) has converted a competitive industry into a monopoly. So production methods and the type of products produced have changed immeasurably in the last hundred years. What is remarkable is that this is not true of Economics journals. Yes, the process of going from accepted manuscript to published copy has been transformed in the past fifteen years or so by the use of computerized technology. However, the editorial process is largely the same as that operating a century ago: manuscripts are submitted, reviewed by the editor and his referees, and, if deemed suitable, an iterative process converges on a publishable article that is printed in hard copy form. If there has been innovation, it has taken the form of the type of articles that Economics journals are publishing today: the range of subjects that the editors of the quality journals are willing to consider seriously for publication is considerably wider than it was fifty years ago. But this merely reflects the changes that have taken place within the profession and the journals cannot be said to have lead the profession in this. (7)

It is often forecast that the future will see drastic changes in the manner in which information is conveyed to scholars. Some predict the current hard bound copy of journals will be not be supplemented with but replaced entirely by information that can be retrieved electronically and printed at the wishes of the reader. The journal system as we have known it for over a hundred years will cease. It would be foolhardy to deny that possibility, but at the same time I am not confident of this outcome. Although these other methods of information retrieval may well partially replace the hard copy, I also see enduring advantages for the conventional form of journal so that the industry we are observing will retain its current features for still many years.

APPENDIX

The purpose of this appendix is to document the statement in the text about the pattern of citations. Inferences from the citation indices about the impact of journals differ from index to index but, for the points I want to make, these differences are for the most part of second-order of importance. I prefer to concentrate here on one such index and discuss it rather than cloud the discussion with several such indices. The work described here was undertaken in the early 1990s.

The citation index I present here was constructed as follows. I arbitrarily selected a set of eleven principal economics journals. I label these the Gold Medal journals. Though the choice of these is arbitrary, I think there would be wide agreement among economists about the inclusion of most of these journals in the set of Gold Medal journals. (8) For each and every journal--not just the Gold Medal journals, but all journals whose contents were listed in the Journal of Economic Literature--we counted the number of times articles published in the Gold Medal journals in 1988 cited articles published in each other journal over the previous four years.

Consider the Journal of Industrial Economics as an example. We counted the number of times that articles published in the Gold Medal journals in 1988 cited articles published in the Journal of Industrial Economics in the years from 1984 to 1987. The answer is, in fact, 26.

We then expressed this number as a percentage of the number of articles published annually by the journal under consideration. Thus if the Journal of Industrial Economics published 30 articles per year, the citation index would be 87 (that is, 100(26/30)). This is a measure of the impact of the Journal of Industrial Economics per article it publishes.

We constructed the values of this index for each of the 320 journals established before 1987 and whose contents were listed in the Journal of Economic Literature. The mean value of this index was 12.15 with a standard deviation of 43.16. Its minimum value was zero and 74% of the 320 journals had a value of zero! Its maximum value was 402 (Econometrica).

What is remarkable is the degree of inequality in citations. A common graphical representation of this is the Lorenz Curve. Suppose we order the journals from lowest in the rankings of citations to highest. Then plot the cumulative percentage of journals on the horizontal axis (ranging from zero to 100%) and the cumulative percentage of citations on the vertical axis. If each journal had the same number of citations, the Lorenz Curve would simply be the diagonal line running from the bottom left hand corner to the upper right hand corner; this would be the representation corresponding to complete equality. But if the bottom ranked journals receive fewer citations while the top ranked journals receive a disproportionate share of citations, then the Lorenz Curve will lie below the diagonal. Such is the case, in fact, and, indeed, its departure from complete equality is marked. Some points along the curve might be instructive.

Denote by y the cumulative percentage share of citations (the vertical axis of the graph) and by x the percentage share of the population of journals (the horizontal axis). Then some points on the Lorenz Curve are as follows:

x 74 75 80 85 90 95 99 100

y 0 0.2 1.5 4.8 12.7 29.8 72.9 100

In other words, 74% of the journals contributed zero percent of all the citations; 80% of the journals contributed 1.5% of all the citations; and 99% of the journals contributed almost 73% of the citations. Or, expressed differently, 26% (100 minus 74) of the journals contributed 100% of the citations, 10% of the journals contributed 87.3% of the citations, 1% of the journals contributed 27.1% of the citations. This gives the impression of substantial inequality among the journals in citations. (9) If a Lorenz Curve with the appearance of that graphed applied to the distribution of incomes in a society--the use to which the Lorenz Curve is often put--that society would be regarded as deeply inegalitarian.

However, just to illustrate Disraeli's maxim, suppose these same data are used to construct another measure of concentration, namely, the Herfindahl index. Let [s.sub.j] be journal j's citations expressed as a fraction of all the journals' citations. The Herfindahl index, H, is defined as [[[summation].sub.i][(s.sub.i)].sup.2]. Clearly H ranges from zero to unity. The minimum value of H is 1/n where n is the number of journals--here 320. The maximum value of H is unity when one journal accounts for all the publications, a monopoly of citations. Larger values of H indicate greater concentration of citations among fewer journals. George Stigler (10) reported values of H of .331 for the auto industry in 1964, of .221 for the cigarette industry in 1963, of .253 for the soap industry in 1958, and of .019 for the fire and casualty insurance companies in 1963. For this sample of 320 journals, I calculate the Herfindahl index to be .044. I am assured a value of the Herfindahl index of .044 would not prompt litigation by the Antitrust Division of the Justice Department! However, note that Stigler's indices relate to the production of U.S. firms only. The Herfindahl index I computed for Economics citations embraces the production of citations internationally. This issue of inequality in the impact of Economics Journals merits further analysis.

John Pencavel (1)

(1) Levin Professor of Economics, Department of Economics, Stanford University, Stanford, California 94305-6072.

(2) See Joshua S. Gans and George B. Shepherd, "How Are the Mighty Fallen: Rejected Classic Articles by Leading Economists", Journal of Economic Perspectives, Vol. 8, No. 1, Winter 1994, 165-79.

(3) Of course, publication in the major journals does not mean the scholar's argument is valid; it means it has survived the reviewers' sceptical examination and deserves to be the subject of critical assessment by a wider audience.

(4) The JEL was the first major Economics journal that subscribers could choose to receive as a CD-Rom rather than in printed, hard copy, form. I started this CD-Rom version of the Journal in March 1995.

(5) The day-to-day administration of EconLit was overseen in the Pittsburgh office of the JEL by Drucilla Ekwurzel who does a splendid job sustaining and nurturing EconLit.

(6) I relegate to an appendix the more technical aspects of one such index.

(7) Perhaps a qualification is in order here with regard to the Journal of Political Economy. It has been willing to publish articles on unconventional research topics though these articles used highly conventional research methods.

(8) These journals were as follows: American Economic Review, Econometrica, Economic Journal, Economica, International Economic Review, Journal of Economic Literature, Journal of Economic Theory, Journal of Political Economy, Quarterly Journal of Economics, Review of Economic Studies, and Review of Economics and Statistics.

(9) Dr. Eugene Garfield of the Institute for Scientific Information has shown me comparable degrees of inequality in other disciplines. Such inequality in economics is also suggested by Arthur M. Diamond's analysis, "The Core Journals of Economics", Journal of Citation Studies, 50, Part 1, January 1989, pp. 2-9.

(10) The Economic Effects of the Antitrust Laws, Journal of Law and Economics, Volume IX, October 1966, pp. 225-58.