Why Peer Review is the Worst Form of Quality Control and Credentialing
Except All Those Other Forms that Have Been Tried From Time to Time
Author’s Note: The Association of American University Presses (AAUP) recently published a Best Practices for Peer Review Handbook. The result of a two-year consensus-building (and peer reviewed) effort by the organization and a subcommittee of seasoned acquiring editors, the 26-page booklet articulates a set of practices that constitute a rigorous peer review process for academic book publishers. Sections of the book include: Why Peer Review is important; The Acquiring Editor’s Choices about Why, When, and How to Conduct Peer Review; Selecting Peer reviewers; Sharing Peer Reviews With Authors; and Peer Reviews as Documents of Record. You can download a Creative Commons licensed edition of the Handbook at: http://www.aaupnet.org/resources/for-members/handbooks-and-toolkits/peer-review-best-practices. — MGD
I frequently make a Big Deal about our capacity and competence with the peer review process for the books that we publish. And I recently had the opportunity to put my mouth where my money is when I helped craft a Best Practices for Peer Review Handbook (see http://bit.ly/1TXsDaz) for the Association of American University Presses (AAUP). I’d like to share some thoughts on the motivation behind that handbook (my thoughts, which are not necessarily the AAUP Board’s thoughts nor those of the AAUP Acquisitions Committee that drafted the Handbook).
What follows, then, is one editor’s reflection on Peer Review’s past,1 present, and future, as revealed through the decision to publish a Best Practice Handbook. My thoughts reflect my world of book2 publishing in the humanities and social sciences, though some of the “macro” phenomena in play here certainly apply across the academy.
So why did AAUP, after 70-plus years decide that they needed to research and publish these fundamental guidelines for peer review best practice? I suspect it comes down to the simultaneous expansion and adaptation of our scholarly publishing landscape. This ongoing transition is an oft-told tale. As institutional support for scholarship (especially scholarship’s publication) dwindles, and as “conventional” markets for cost recovery (book sales) also wither on the vine, scholarly presses are exploring new models for dissemination and cost recovery. On a related track, academic institutions and their funders (public and private) are seeking ways to have research they feel they have already funded more broadly accessible without fees or other barriers to all readers/consumers (Open Access). Publishers, therefore, are experimenting with “flipped” publishing models, where the costs of publication are paid upfront by producers rather than consumers of the works (costs that include overhead for the entire publishing project, the print and bind cost for a book version of a project is a pretty small percentage of the whole).
Interestingly, at the same time as these economic and technological changes are taking place, university presses are publishing more books than ever3. And membership in the AAUP is expanding. There are new university presses emerging4 as top-flight universities revisit the “value add” of a focused, reputable university press that can expand their capacity for research, teaching, service, and, yes, their “brand.”
All of this churning has presented challenges, to be sure, but it has also produced opportunities. I mentioned above that there are some new university presses emerging. Add to that the growth of library publishing initiatives, as well as government and professional organizations lifting their information dissemination game.
Which brings me back to the AAUP. I think it’s safe to say that the AAUP regards itself as a “big tent” organization, encouraging and recruiting fellow travelers (or fellow campers, perhaps) — sometimes as full-fledged members, sometimes associate members, and sometimes just peers working on a shared set of activities, such as getting work that edifies in front of readers who wish to be edified.
So as the organization works on exploring new partnerships, it also needed to define what the “core competencies” of a good university press might be. The AAUP’s current guidelines for full membership say a press, “must have a committee or board of the faculty (or equivalent, if the press is not affiliated with a university) that certifies the scholarly quality of the books published through peer review consistent with commonly understood notions of peer review.”
Which begs the question, “what are our commonly understood notions of peer review?” That is what our acquisitions editor committee tried to find out. I won’t go into the details of where we landed regarding commonly understood notions, but those who visit the handbook will see that we were aware of a pretty diverse set of practices. As the report explains, “the peer review process is highly complex, involves many individuals, and must be responsive to the norms of the appropriate fields.”5
But, again, this was a broad brush look at best practice. There is a lot of the “art” of acquisitions as it pertains to peer review that we did not have the pages to explore fully. As an example, in the section on choosing appropriate peer reviewers, we foregrounded a reader’s potential to judge the scholarship/argument/presentation of a work. But we could have supplemented that section with more discussion of diversity, identity, and balance. Gender, race, class, disability, sexuality, and other categories and identities are a significant part of the more nuanced decisions and considerations that editors and their advisers think through as they manage peer review — more so in some disciplines than others.
The AAUP handbook joins an ongoing, vigorous discussion about the importance, proper execution, and assorted flaws of peer review. I would hate to think that some readers may see the Best Practice Handbook as a “rear-guard” action, defending the academic press world from hordes of charlatan invaders. In addition to striving for a “best practice” that secures membership and reassures the scholarly ecosystem, university presses are also eager to experiment with alternate models for evaluating and strengthening good scholarship.6 What these discussions hold for the future is hard to say. We have been discussing new measures for credentialing scholarship and for disseminating scholarship for all of the 27 years I have worked in publishing. I will note here that the conversation has become more global (another source of the AAUP’s growth), and the cohort of publishers working with (or within) academic institutions is becoming ever more connected. All positive signs for innovation and improved practices, I’d say. So, the conversation continues and it is my hope that the AAUP Handbook serves as a helpful catalyst for that conversation as well as a “baseline” for scholars, administrators, and institutions that support scholarly presses.
One of my favorite “inspirational” quotes that I think describes quite well the university press world comes from John Gardner (Secretary of Health, Education, and Welfare under President Lyndon Johnson): “The society which scorns excellence in plumbing as a humble activity and tolerates shoddiness in philosophy because it is an exalted activity will have neither good plumbing nor good philosophy: neither its pipes nor its theories will hold water.” This simple truth reminds me that we must cultivate the very best ideas, test and re-test those ideas (peer review), and maintain the very best “pipes” to disseminate those same ideas as broadly and cost effectively as possible (books, eBooks, Websites, blogs, apps) to a readership that remains eager to learn.
It is my view that the ideas, the pipes, and the learning all require financial support. We are plumbers and philosophers all.
- For some background on peer review, Trevor Lipscombe wrote a marvelous essay on the sectarian origins of peer review and how that has trickled down to the present day (“Burn this Article” – see http://muse.jhu.edu/article/613577). And I describe elsewhere (see https://ugapress.wordpress.com/2016/06/16/peering-into-the-dark-underbelly-of-peer-review-or-practice-makes-best/) our committee’s own peer review and drafting process that produced the handbook.
- I’ve had some experience with online publishing of digital scholarship, but that remains more experimental to-date and our Best Practice Handbook focused on more established book conventions.
- Based on reported numbers from AAUP Annual Directory of Presses, 2000 through 2015. See also Crossick, Geoffrey. “Monographs and Open Access: A report to HEFCE.” Higher Education Funding Council for England (HEFCE), January 2015, p. 21, which reports title output of monographs among the four biggest academic presses as doubling between 2004 and 2013.
- The number of new presses is small, in North America at least. I know of at least two new Presses in the past couple of years, with at least two more in the start-up phase (some have not announced publicly).
- AAUP. Best Practices for Peer Review. 2016, p. 6.
- For example, Claire Potter at the New School is in the midst of an experiment with UNC Press, writing her next book in a shared environment (see http://digitalulab.org/2016/06/05/why-blog-a-book/) that allows ongoing comments as she writes and rewrites about the future of digital scholarship.