Who's Afraid of Peer Review?

"Who's Afraid of Peer Review?" is an article written by Science correspondent John Bohannon that describes his investigation of peer review among fee-charging open-access journals. Between January and August 2013, Bohannon submitted fake scientific papers to 304 journals owned by as many fee-charging open access publishers. The papers, writes Bohannon, "were designed with such grave and obvious scientific flaws that they should have been rejected immediately by editors and peer reviewers", but 60% of the journals accepted them. The article and associated data were published in the 4 October 2013 issue of Science as open access.[2][3]

Letharia vulpina closeup
Letharia vulpina, one of the species of lichen credited with having a cancer-inhibiting molecule in a fake manuscript.[1]


The first fee-charging open access scientific journals began appearing in 2000 with the creation of BioMed Central and then the Public Library of Science. Rather than deriving at least some of their revenue from subscription fees, fee-charging open access journals only charge the authors (or their funders) a publication fee. The published papers are then freely available on the internet. This business model, gold open access, is one of several solutions devised to make open access publishing sustainable.[4] The number of articles published open access, or made freely available after some time behind a paywall (delayed open access), has grown rapidly. In 2013 more than half of the scientific papers published in 2011 were available for free.[5]

In part because of the low barrier to entry into this market, as well as the fast and potentially large return on investment, many so-called "predatory publishers" have created low-quality journals that provide little to no peer review or editorial control, essentially publishing every submitted article as long as the publication fee is paid. Some of these publishers additionally deceive authors about publication fees, use the names of scientists as editors and reviewers without their knowledge, and/or obfuscate the true location and identity of the publishers.[6] The prevalence of these deceptive publishers, and what the scientific community should do about them, has been hotly debated.[7]


Fake papers

Bohannon used Python to create a "scientific version of Mad Libs".[2][8] The paper's template is "Molecule X from lichen species Y inhibits the growth of cancer cell Z". He created a database of molecules, lichens, and cancer cells to substitute for X, Y, and Z. The data and conclusions were identical in every paper. The authors and their affiliations were also unique, and fake. The papers all described the discovery of a new cancer drug extracted from a lichen, but the data did not support that conclusion and the papers had intentionally obvious flaws.[9][10]

Publisher targets

To build a comprehensive list of fee-charging open access publishers, Bohannon relied on two sources: Beall's List of predatory publishers and the Directory of Open Access Journals (DOAJ). After filtering both lists for open access journals published in English, that charge authors a publication fee, and that have at least one medical, biological, or chemical journal, the list of targets included 304 publishers: 167 from the DOAJ, 121 from Beall's list, and 16 that were listed by both. The investigation focused entirely on fee-charging open access journals. Bohannon did not include other types of open access journals or subscription journals for comparison because the turnaround time for reviews in traditional journals is too long. The study consequently makes no claim about the relative quality of the different types of journals.[11]


Acceptance versus rejection

In total, 157 of the journals accepted the paper and 98 rejected it, with the other 49 not having completed their evaluation by the time Bohannon wrote his article.[2] Of the 255 papers that underwent the entire peer review process to acceptance or rejection, about 60% of the final decisions occurred with no sign of actual peer review. For rejections, that may possibly have reflected filtering at the editorial level, but for acceptance can only reflect a flawed process. Only 36 submissions generated review comments recognizing any of the paper's scientific problems. 16 of those 36 papers were nonetheless accepted, in spite of poor to damning reviews. Many of the journals that accepted the paper are published by prestigious institutions and publishing companies, including Elsevier, Sage, Wolters Kluwer (through its subsidiary Medknow), and several universities.[2]

Among those that rejected the paper are journals published by PLOS, BioMed Central, and Hindawi. The peer review provided by PLOS ONE was reported to be the most rigorous of all, and it was the only journal that identified the paper's ethical problems, for example the lack of documentation of how animals were treated in the creation of the cancer cell lines.[2]

DOAJ versus Beall's list

Among the publishers on Beall's list that completed the review process, 82% accepted the paper. Bohannon stated "the results show that Beall is good at spotting publishers with poor quality control". According to Jeffrey Beall, who created the list, this supports his claim to be identifying "predatory" publishers.[12] However, the remaining 18% of publishers identified by Beall as predatory rejected the fake paper, causing science communicator Phil Davis to state, "That means that Beall is falsely accusing nearly one in five".[13]

Among the DOAJ publishers that completed the review process, 45% accepted the paper.[13] According to a statement published on the DOAJ website, new criteria for inclusion in the DOAJ are being implemented.[14]

Global map of journal fraud

Along with the report, Science published a map that shows the location of publishers, editors, and their bank accounts, color-coded by acceptance or rejection of the paper. The locations were derived from IP address traces within the raw headers of e-mails, WHOIS registrations, and bank invoices for publication fees. India emerged as the world's largest base for fee-charging open-access publishing, with 64 accepting the fatally flawed papers and only 15 rejecting it.[2] The United States is the next largest base, with 29 publishers accepting the paper and 26 rejecting it. In Africa, Nigeria has the largest number, of which 100% accepted the paper.[15]


Responses from the open-access academic publishing industry

Since the story was released, publishers of three journals have stated that they are shutting them down.[16] The DOAJ is reviewing its list and instituting tighter criteria for inclusion.[17] The Open Access Scholarly Publishers Association (OASPA) formed a committee to investigate the circumstances that led to the acceptance of the fake paper by three of its members.[18] On 11 November 2013, OASPA terminated the membership of two publishers (Dove Medical Press and Hikari Ltd.) who accepted the fake paper. Sage Publications, which also accepted a fake paper, was put "under review" for 6 months.[19] Sage announced in a statement that it was reviewing the journal that accepted the fake paper, but that it would not shut it down.[20] Sage's membership was reinstated at the end of the review period following changes to the journal's editorial processes.[21]

Responses from the scientific community

Within hours of its publication, the Science investigation came under intense criticism by some supporters of the open-access movement.[22][23]

The first substantial critique was posted by PLOS cofounder Michael Eisen on his blog. "To suggest – as Science (though not Bohannon) are trying to do – that the problem with scientific publishing is that open access enables internet scamming is like saying that the problem with the international finance system is that it enables Nigerian wire transfer scams. There are deep problems with science publishing. But the way to fix this is not to curtail open-access publishing. It is to fix peer review."[24] Eisen pointed out the irony of a subscription-based journal like Science publishing this report when its own peer review has failed so badly before, as in the 2010 publication of the arsenic DNA paper.

In an exchange between Eisen and Bohannon in a discussion hosted by Peter Suber, director of the Harvard Open Access Project, Eisen criticized the investigation for the bad publicity it generated for the open-access movement.[25] "Your study exclusively targeted open access journals – [which] strongly suggested, whether you meant to suggest this or not, that open access journals are more likely to engage in shoddy peer review and therefore more deserving of scrutiny." Bohannon responded that this critique was equivalent to "shooting the messenger".

There have also been many statements of support for the investigation,[26][27] and statements of concern about the publishing fraud that it revealed.[28][29][30] The Committee on Publication Ethics has responded that "There is no doubt that this 'sting' raises a number of issues ... though I'd argue they are not necessarily the ones that Science thinks are top priorities."[31]


Some scientists have discussed a number of options for making peer review more transparent.[32] Doing so would make it harder to maintain a predatory journal that does no peer review, because the record of peer review would be lacking or would need to be faked.[33] Another option is to more rigorously vet journals, for example by further empowering DOAJ or OASPA. DOAJ has recently tightened up its inclusion criteria, with the purpose of serving as a whitelist, very much like Beall's has been a blacklist.[34]

See also


  1. ^ Paper 1 in the data supplement for Bohannon 2013.
  2. ^ a b c d e f Bohannon, John (2013). "Who's Afraid of Peer Review?". Science. 342 (6154): 60–65. doi:10.1126/science.342.6154.60. PMID 24092725. Retrieved 20 October 2013.
  3. ^ Bohannon, John (2013). "Who's Afraid of Peer Review: Data and Documents". Science. 342 (6154): 60–5. doi:10.1126/science.342.6154.60. PMID 24092725.
  4. ^ Suber, Peter. "Open Access Overview". Earlham College. Retrieved 21 October 2013.
  5. ^ Van Noorden, Richard (20 August 2013). "Half of 2011 papers now free to read". Nature. 500 (7463): 386–387. doi:10.1038/500386a. ISSN 0028-0836. PMID 23969438.
  6. ^ Knox, Richard (3 October 2013). "Some Online Journals Will Publish Fake Science, For A Fee". NPR. Retrieved 20 October 2013.
  7. ^ Butler, Declan (27 March 2013). "Investigating journals: The dark side of publishing". Nature. 495 (7442): 433–435. doi:10.1038/495433a. ISSN 0028-0836. PMID 23538810.
  8. ^ Koebler, Jason. "Inside Science Magazine's 'Sting' of Open Access Journals". Motherboard. Retrieved 22 October 2013.
  9. ^ "Science's Sokal moment". The Economist. 5 October 2013. Retrieved 20 October 2013.
  10. ^ Vergano, Dan (3 October 2013). "Fake Cancer Study Spotlights Bogus Science Journals". National Geographic. Retrieved 22 October 2013.
  11. ^ Oransky, Ivan. "Science reporter spoofs hundreds of open access journals with fake papers". Retraction Watch. Retrieved 22 October 2013.
  12. ^ Beall, Jeffrey. "Science Magazine Conducts Sting Operation on OA Publishers". Scholarly Open Access. Archived from the original on 12 October 2013. Retrieved 22 October 2013.
  13. ^ a b Davis, Phil. "Open Access "Sting" Reveals Deception, Missed Opportunities". The Scholarly Kitchen.
  14. ^ "DOAJ's response to the recent article in Science entitled "Who's Afraid of Peer Review?"". Directory of Open Access Journals. Archived from the original on 23 October 2013. Retrieved 22 October 2013.
  15. ^ "Peer review map". Science. AAAS. Retrieved 21 October 2013.
  16. ^ Oransky, Ivan (17 October 2013). "Fallout from Science's publisher sting: Journal closes in Croatia". Retraction Watch.
  17. ^ "Second response to the Bohannon article". Directory of Open Access Journals. Archived from the original on 10 November 2013. Retrieved 21 October 2013.
  18. ^ Redhead, Claire. "OASPA's response to the recent article in Science entitled "Who's Afraid of Peer Review?"". Open Access Scholarly Publishers Association. Retrieved 21 October 2013.
  19. ^ "OASPA's second statement following the article in Science entitled "Who's Afraid of Peer Review?"". Open Access Scholarly Publishers Association. Retrieved 11 November 2013.
  20. ^ Gamboa, Camille. "Statement by SAGE on the Journal of International Medical Research". Sage. Retrieved 22 November 2013.
  21. ^ Shaffi, Sarah (29 April 2014). "OASPA reinstates Sage membership". The Bookseller. Retrieved 2 June 2014.
  22. ^ Basken, Paul (4 October 2013). "Critics Say Sting on Open-Access Journals Misses Larger Point". The Chronicle of Higher Education. Retrieved 20 October 2013.
  23. ^ Davis, Philip. "Post Open Access Sting: An Interview With John Bohannon". The Scholarly Kitchen. Retrieved 12 November 2013.
  24. ^ Eisen, Michael. "I confess, I wrote the Arsenic DNA paper to expose flaws in peer-review at subscription based journals". Retrieved 21 October 2013.
  25. ^ Peter Suber. "New "sting" of weak open-access journals".
  26. ^ Yirka, Bob. "Research paper publishing sting reveals lax standards of many open-access journals". Phys.org. Retrieved 20 October 2013.
  27. ^ Davis, Phil. "Open Access "Sting" Reveals Deception, Missed Opportunities". The Scholarly Kitchen. Retrieved 20 October 2013.
  28. ^ Tatalović, Mićo. "Sting exposes 'wild west' of open-access publishing". Scidev.net. Retrieved 20 October 2013.
  29. ^ Mudur, G S (3 October 2013). "Dubious journal fear stalks India". The Telegraph. Retrieved 20 October 2013.
  30. ^ Shieber, Stuart. "Lessons from the faux journal investigation". Harvard Law School. Retrieved 20 October 2013.
  31. ^ "COPE response to Science paper submission of fake paper, by Virginia Barbour, on behalf of COPE council".
  32. ^ "Transparency in peer review". Nature Materials. Nature. 10 (2): 81. 2011. doi:10.1038/nmat2952. PMID 21258345. Retrieved 21 October 2013.
  33. ^ "Is this peer reviewed? Predatory journals and the transparency of peer review".
  34. ^ Van Noorden, R. (2014). "Open-access website gets tough". Nature. 512 (7512): 17. doi:10.1038/512017a. PMID 25100463.

External links

Beall's List

Beall's List was a list of predatory open-access publishers that was maintained by University of Colorado librarian Jeffrey Beall on his blog Scholarly Open Access. The list aimed to document open-access publishers who did not perform real peer review, effectively publishing any article as long as the authors pay the open access fee. Originally started as a personal endeavor in 2008, Beall's List became a widely followed piece of work by the mid 2010s. Its influence led some publishers on the list to threaten defamation lawsuits against Beall, as well as lodge official complaints against Beall's work to the University of Colorado. As a result, Beall deactivated his blog and the list in January 2017.

The closure of Beall's List was cited by some as a tragedy, and successors have set out to continue Beall's work.

Bentham Science Publishers

Bentham Science Publishers is a company that publishes scientific, technical, and medical journals and e-books. It publishes 140 subscription-based academic journals and over 60 open access journals. It is based at Sharjah in the United Arab Emirates, and has operating units in the United States, Japan, China, India, and the Netherlands. The company workforce is outsourced to China, India, Japan and Pakistan.Bentham Open, its open-access branch, has received some criticism for its questionable peer-review practices, and was listed as a "potential, possible, or probable predatory scholarly open-access publisher" in Jeffrey Beall's list of Predatory Publishers.

Dove Medical Press

Dove Medical Press is an academic publisher of open access peer-reviewed scientific and medical journals, with offices in Manchester, London (United Kingdom), Princeton, New Jersey (United States), and Auckland (New Zealand).In September 2017, Dove Medical Press was acquired by the Taylor and Francis Group (Informa PLC).As an open access publisher, Dove charges a publication fee to authors or their institutions or funders. This charge allows Dove to recover its editorial and production costs and to create a pool of funds that can be used to provide fee waivers for authors from lesser developed countries. Articles published are available via an interface following the Open Archives Initiative Protocol for Metadata Harvesting, a set of uniform standards promulgated by the Open Archives Initiative allowing metadata on archive holdings.Dove is a member of the Association of Learned and Professional Society Publishers, the Committee on Publication Ethics, and the Open Archives Initiative. As of September 2016, it publishes over 100 journals.

E-Century Publishing Corporation

e-Century Publishing Corporation is a publisher of seventeen open access scientific journals based in Madison, Wisconsin. Four of them are indexed in the Science Citation Index Expanded: the American Journal of Translational Research, the American Journal of Cancer Research, the International Journal of Clinical and Experimental Medicine, and the International Journal of Clinical and Experimental Pathology. It has been criticised for its role in publishing plagiarised work, falsified data and duplicate publication.One of its journals — the American Journal of Nuclear Medicine and Molecular Imaging — was targeted in the Who's Afraid of Peer Review? sting operation and rejected the fake paper.


Elsevier (Dutch pronunciation: [ˈɛlzəviːr]) is a Dutch information and analytics company and one of the world's major providers of scientific, technical, and medical information. It was established in 1880 as a publishing company. It is a part of the RELX Group, known until 2015 as Reed Elsevier. Its products include journals such as The Lancet and Cell, the ScienceDirect collection of electronic journals, the Trends and Current Opinion series of journals, the online citation database Scopus, and the ClinicalKey solution for clinicians. Elsevier's products and services include the entire academic research lifecycle, including software and data-management, instruction and assessment tools.Elsevier publishes more than 430,000 articles annually in 2,500 journals. Its archives contain over 13 million documents and 30,000 e-books. Total yearly downloads amount to more than 900 million.Elsevier's high operating profit margins (37% in 2017) and its copyright practices have subjected it to criticism by researchers.

Hindawi Publishing Corporation

Hindawi Publishing Corporation is a commercial publisher of scientific, technical, and medical (STM) literature. Founded in 1997, Hindawi currently publishes more than 250 peer-reviewed scientific journals as well as a number of scholarly monographs, with an annual output of roughly 20,000 articles each year. As of October 2014, 11% of its journals were indexed in the Science Citation Index Expanded. The company has its headquarters in London, an office in Cairo and a virtual office address in New York City. Since 2007, all of Hindawi's journals have been open access and published under a Creative Commons Attribution License (CC-BY). Hindawi has been criticized for some of its editorial practices – including an alleged reliance on staff vetting of submissions rather than peer review by academics – and for its use of email spam in soliciting editorial board memberships and manuscripts. It has been described as a borderline predatory publisher.

Jeffrey Beall

Jeffrey Beall is an American librarian, best known for drawing attention to "predatory open access publishing", a term he coined, and for creating what is now widely known as Beall's list, a list of potentially predatory open-access publishers.

He is a critic of the open access publishing movement and is especially known for his blog, Scholarly Open Access.

He has also written on this topic in The Charleston Advisor, in Nature, in Learned Publishing, and elsewhere.When Beall created his list, he was employed at the University of Colorado, Denver. More recently, he was a librarian at Auraria Library in Denver until March 2018.

John Bohannon

John Bohannon is Director of Science at Primer, an artificial intelligence company headquartered in San Francisco, California. He is known for his career prior to Primer as a science journalist and Harvard University biologist, most notably with his “Gonzo Scientist” online series at Science Magazine and his creation of the annual “Dance Your PhD” contest. His investigative journalism work includes:

critiquing the Lancet surveys of Iraq War casualties (2006),

uncovering serious problems with the peer review process at a large number of journals that charge fees to authors (2013), and

showing how uncritical mass media can be of claims made in fake scientific papers (2015).

List of scholarly publishing stings

This is a list of scholarly publishing "sting operations" such as the Sokal affair. These are nonsense papers that were accepted by an academic journal or academic conference; the list does not include cases of scientific misconduct. The intent of such publications is typically to expose shortcomings in a journal's peer review process or to criticize the standards of pay-to-publish journals.


MDPI is an organisational acronym used by two related organisations, Molecular Diversity Preservation International and Multidisciplinary Digital Publishing Institute, which were both co-founded by Shu-Kun Lin. The first organisation, Molecular Diversity Preservation International, founded in 1996, is primarily a chemical sample archive, with some scholarly publishing and conference activities. The second organisation, Multidisciplinary Digital Publishing Institute, was founded in 2010, primarily as a publisher. As of 2018 MDPI publishes 213 academic journals, including 37 that have received an impact factor. However, the quality of MDPI's peer review is disputed.MDPI was included on Jeffrey Beall's list of predatory open access publishing companies in 2014 and was removed in 2015. Beall's list was shut down in 2017; Beall later wrote that he had been pressured to shut down the list by various publishers, specifically mentioning MDPI. The publisher was downgraded to level 0, the lowest level, in the Norwegian Scientific Index for the year 2019.

Medknow Publications

Medknow Publications also known as Wolters Kluwer Medknow or simply Medknow, is a publisher of academic journals on behalf of learned societies and associations. Previously an independent Indian publisher, Medknow is now part of within Wolters Kluwer's Health Division, and is part of Wolters Kluwer India.

Medknow is not a member of the Open Access Scholarly Publishers Association. Some, but not all of its journals are listed in the Directory of Open Access Journals. The publisher briefly appeared on Beall's list in 2011, but was taken down, being described as "[a] publisher for many well-respected Indian professional societies and is disseminating abundant, high-quality research."

OMICS Publishing Group

OMICS Publishing Group is a predatory publisher of open access journals. It started publishing its first journal in 2008. By 2015, it claimed over 700 journals, although about half of them were defunct.

Its subsidiaries include iMedPub LTD and Conference Series LLC LTD. Other organisations linked to OMICS are EuroSciCon Ltd, Allied Academies, Trade Science Inc, and Meetings International.OMICS has come under attack by numerous academics and the United States government over the validity of the peer review by OMICS journals, the appropriateness of its fees and marketing, and the apparent advertising of the names of scientists as journal editors or conference speakers without their knowledge or permission. The U.S. National Institutes of Health sent a cease-and-desist letter to OMICS in 2013, demanding it to discontinue with false claims of affiliation with U.S. government entities or employees. In August 2016 OMICS became the first academic publisher to be sued by the U.S. Federal Trade Commission (FTC), for deceptive practices.

OMICS has responded to criticisms by avowing a commitment to open access publishing, claiming that detractors are traditional subscription-based publishers who feel threatened by their open access publishing model. It responded to the FTC suit by maintaining that their practices were legal and claiming that corporate interests were driving the suit. It has also threatened a prominent critic, Jeffrey Beall, with a US$1 billion lawsuit for defamation.

Open Access Scholarly Publishers Association

The Open Access Scholarly Publishers Association (OASPA) is a non-profit trade association representing the interests of open access journal publishers globally in all scientific, technical and scholarly disciplines. Along with promoting open access publishers (particularly open access journals), OASPA sets best practices and provides a forum for the exchange of information on and experiences of open access. OASPA brings together the major open access publishers on the one hand and independent—often society-based or university-based—publishers on the other, along with some hybrid open access publishers. While having started out with an exclusive focus on open access journals, it is now expanding its activities to include matters pertaining to open access books too.

Open access

Open access (OA) is a mechanism by which research outputs are distributed online, free of cost or other barriers, and, in its most precise meaning, with the addition of an open license applied to promote reuse.Academic articles (as historically seen in print-based academic journals) have been the main focus of the movement. Conventional (non-open access) journals cover publishing costs through access tolls such as subscriptions, site licenses or pay-per-view charges. Open access can be applied to all forms of published research output, including peer-reviewed and non peer-reviewed academic journal articles, conference papers, theses, book chapters, and monographs.

Predatory open-access publishing

Predatory open-access publishing, or sometimes write-only publishing, is an exploitative open-access academic publishing business model that involves charging publication fees to authors without providing the editorial and publishing services associated with legitimate journals (open access or not). The idea that they are "predatory" is based on the view that academics are tricked into publishing with them, though some authors may be aware that the journal is poor quality or even fraudulent. New scholars from developing countries are said to be especially at risk of being misled by predatory practices."Beall's List", a report that was regularly updated by Jeffrey Beall of the University of Colorado until January 2017, set forth criteria for categorizing publications as predatory. The list was taken offline by the author in January 2017. A demand by Frontiers Media to open a misconduct case against Beall was reported as the reason Beall closed the list, but an investigation by the university was closed with no findings. After the closure, other efforts to identify predatory publishing have sprouted, such as the paywalled Cabell's blacklist, as well as other lists (some based on the original listing by Beall).


SCIgen is a computer program that uses context-free grammar to randomly generate nonsense in the form of computer science research papers. All elements of the papers are formed, including graphs, diagrams, and citations. Created by scientists at the Massachusetts Institute of Technology, its stated aim is "to maximize amusement, rather than coherence."

Scholarly peer review

Scholarly peer review (also known as refereeing) is the process of subjecting an author's scholarly work, research, or ideas to the scrutiny of others who are experts in the same field, before a paper describing this work is published in a journal, conference proceedings or as a book. The peer review helps the publisher (that is, the editor-in-chief, the editorial board or the program committee) decide whether the work should be accepted, considered acceptable with revisions, or rejected.

Peer review requires a community of experts in a given (and often narrowly defined) field, who are qualified and able to perform reasonably impartial review. Impartial review, especially of work in less narrowly defined or inter-disciplinary fields, may be difficult to accomplish, and the significance (good or bad) of an idea may never be widely appreciated among its contemporaries. Peer review is generally considered necessary to academic quality and is used in most major scholarly journals, but it by no means prevents publication of invalid research. Traditionally, peer reviewers have been anonymous, but there is currently a significant amount of open peer review, where the comments are visible to readers, generally with the identities of the peer reviewers disclosed as well.

Timeline of the open-access movement

The following is a timeline of the international movement for open access to scholarly communication.

World Academy of Science, Engineering and Technology

The World Academy of Science, Engineering and Technology or WASET is a publisher of a number of open access journals on a wide variety of scientific and technical subjects. The publisher has been listed as a "potential, possible, or probable" predatory publisher by Jeffrey Beall and is listed as such by the Max Planck Society.

This page is based on a Wikipedia article written by authors (here).
Text is available under the CC BY-SA 3.0 license; additional terms may apply.
Images, videos and audio are available under their respective licenses.