ピア・レビュー／Peer Review_RCR [TEXT]

English Modules

Drafted date: 2016.3.31
Last update: 2016.3.31
Peer Review
< Material provided by >
APRIN, Association for the Promotion of Research Integrity
Contents
Introduction
The Role of the Peer Review
The Peer Review Process
The National Institutes of Health (NIH) Grant Review Process
National Science Foundation (NSF) Grant Review Process
MEXT Grants-in-Aid for Scientific Research Review Process
Ethics and the Peer Review
Confidentiality
Constructive Critique
Competence
Impartiality and Integrity
Disclosure of Conflict of Interest
Timelines and Responsiveness
Potential Problems with Peer Reviews
Resolving Problems with a Review
Improving the Peer Review Process
Conclusion
References
Peer Review
Introduction
P 1/16
The peer review is the evaluation or review of work by experts in the same field (by one’s peers). In scientific research, the evaluation or review of scholarly publications by other experts in the field is widely accepted as the best way to maintain standards of quality and to produce a fair evaluation. In this module, “review” or “evaluation” may be used to refer to a “peer review.”

In both the arts and the sciences, peer-reviewed publications are an essential foundation upon which knowledge is built.

Publishing articles and books allows for the evaluation and discussion of new ideas. It also enables researchers to receive credit and pursue professional advancement. Researchers should recognize that they have many responsibilities relating to being an author or a peer reviewer, and sometimes these responsibilities are viewed in different ways. Through this module, you will learn various kinds of issues that an author or a peer reviewer should be aware of when considering a work’s benefits to society.

Learning Objectives

Through this module, you will be able to:
  • Recognize the importance of responsible peer reviews.
  • Characterize the role of a peer reviewer in assessing a paper or grant application.
  • Examine the major ethical issues surrounding peer review and publication.
  • Describe some of the ways to deal with controversies or conflicts that might arise in the peer review process.
  • Gain familiarity with resources around peer review.
  • Describe the roles of editor and peer reviewer.
Peer Review
The Role of the Peer Review
P 2/16
Today, all major funding agencies require peer reviews of grant applications, and most scholarly journals require them for submitted manuscripts. Professional advancement is often based on the ability to get articles published in quality, peer-reviewed journals.

The Peer Review Process
The mechanism of peer reviews helps to ensure that ideas are scrutinized before they are shared widely with other scholars or the public. Thus, where the quality of the review is high, the review also plays a role in improving the caliber and quality of a grant award application or a publication.

After a manuscript is submitted to a journal, an editor typically requests reviews from members of the journal’s advisory board or from external reviewers with expertise in the subject of the article. Most journals use a single-blind review process, in which the author’s identity is usually known by the reviewer, but the reviewer’s identity is not known by the author. Some journals use a double-blind review process, in which neither the author nor the reviewer knows the other person’s identity. The double-blind review process will be discussed again below.

Peer reviewers are supposed to provide insight into multiple aspects of a manuscript, including the significance of the research and whether it employs an appropriate approach to solve the problem and provides adequate and fair credit to others in the field. Reviewers also should comment on the originality and novelty of the work and whether the research design is adequate to support its conclusions, the interpretation of data is adequate, the reasoning that drives to the conclusion is valid and where there are grammatical errors or unclear writing. These must all be communicated to the editor. In some cases, a reviewer may request the editor not to disclose some of their comments to the author. In general, the editor considers the feedback from one or more reviewers and then makes a determination as to whether the paper should be accepted as is, conditionally accepted stating something like it “will be acceptable if an appropriate revision is made” or rejected.

The peer review of grant applications works somewhat differently from that of papers for publication. For example, when researchers submit a grant application to a national funding agency, the agency will form a committee, often with external reviewers, that evaluates the quality of the application. The various funding agencies differ in how they operate with regard to funded research projects, including how they decide which projects to fund, when they monitor projects for interim assessment and how they evaluate projects prior to publication of the final outcome.
Peer Review
P 3/16
The National Institutes of Health (NIH) Grant Review Process
In the U.S., the National Institutes of Health (the NIH, an agency of the US Department of Health and Human Services that is the American counterpart to the Ministry of Health, Labour and Welfare in Japan) is the primary agency of the federal government responsible for research grants including those for biomedical research. The NIH uses a two-level review process for grant applications1. The first level is conducted by a committee whose members have expertise in the subject of the applications. The second level of review is conducted by an advisory council made up of researchers unaffiliated with the initial committee and lay members of the general public, including those from patient-advocate groups.
Peer Review
P 4/16
National Science Foundation (NSF) Grant Review Process
The National Science Foundation (NSF), another major funding agency providing a monumental amount of research grants comparable to that of the NIH, evaluates grant proposals using two main criteria: intellectual merit and broader impact.2

Issues that reviewers for the NSF are likely to consider include:
  • The proposing researchers’ qualifications.
  • The extent to which a project is creative and original.
  • How the research project would advance discovery, promote teaching and benefit society.
Proposals received by the NSF are reviewed by an NSF program officer and usually an evaluation committee consisting of external reviewers who are experts in the proposal’s field. The findings of these reviewers help to inform the program officer’s recommendations which are then evaluated by senior NSF staff.

Approved NSF grants, which generally last from 1 to 5 years, are reviewed annually by outside experts to assess their progress.
Peer Review
P 5/16
MEXT Grants-in-Aid for Scientific Research Review Process
The Japanese counterparts to the research grants from the NIH (National Institutes of Health) and the NSF (National Science Foundation) in the U.S. are the Grants-in-Aid sponsored by the Ministry of Education, Culture, Sports, Science and Technology (MEXT) and other ministries such as the Ministry of Health, Labour and Welfare (MHLW) and the Ministry of Economy, Trade and Industry (METI). The most well-known competitive research grants are the Grants-in-Aid for Scientific Research (Kakenhi) provided by MEXT, which accounts for 40 percent of the total number of competitive research grants in Japan.

A part of the grant applications submitted to the MEXT Grants-in-Aid for Scientific Research is evaluated by the Japan Society for the Promotion of Science (JSPS). Most of the grants applications are evaluated by a two-level review process similar to that of the NIH. The first level of the process is document review conducted by four to six experts per application, and the second level is conducted based on the results of the document review by a committee that consists of several to about 30 experts depending on the research area. About 6,000 people are registered on the list of referees (reviewers) as experts who enroll in these review processes. To ensure transparency, the list of referees is made public upon completion of all review processes after a certain period of time.

Review processes in the U.S. and in Japan are different in several aspects; one such difference is that the former requires involvement of a small number of people in an extensive way while the latter requires involvement of a large number of people in a non-extensive way. The former procedure is called the “Primary Reviewer” method, in which a main reviewer and a secondary reviewer independently prepare a detailed review report and present them in the joint review meeting of the first level process for the members of the review committee to use as the basis for their discussion. Since about 100 applications are allocated to 18 to 20 committee members in such a way that each application is to be reviewed by one primary and one secondary reviewer, each committee member is put in charge of a relatively small number of applications, i.e., about 5 applications as a primary and another about 5 applications as a secondary reviewer. And a relatively long time can be spent per application. The method employed in Japan, in which a large number of people are involved in the first level of review, is generally considered acceptable in terms of “fairness.” On the other hand, the problems with this method lie in that research projects cannot be assessed so thoroughly as each reviewer evaluates an application in a short period of time, and the opinions of reviewers with low- and high-level expertise are absorbed equally.
Peer Review
Ethics and the Peer Review
P 6/16
Peer reviewers play important roles in the publication process, and ultimately have an impact on authors, publishers and the potential audience for the published work. Peer reviewers must uphold ethical standards in order to maintain the integrity of the review process.

The Council of Science Editors (CSE) outlines the following ethical responsibilities on the part of reviewers and states that these should apply to the publication process in any scholarly field. These responsibilities include3:
  • Confidentiality
  • Constructive critique
  • Competence
  • Impartiality and integrity
  • Disclosure of conflict of interest
  • Timelines and responsiveness
Peer Review
P 7/16
Confidentiality
When submitting manuscripts for publication or sending grant proposals, authors trust reviewers to maintain confidentiality. Manuscripts under review are information considered classified and thus should not be distributed to others without permission. If a need to share some of this knowledge arises, permission must be granted by the editor or the research funding agency. Outside of completing the review, reviewers should not make use of the information gained from a reviewed manuscript or grant proposal: Such use is called “plagiarism” and deemed research misconduct. Furthermore, reviewers are required to destroy copies of manuscripts or grant proposals after the review process concludes.
Peer Review
P 8/16
Constructive Critique
The CSE states as follows: “Reviewer comments should acknowledge positive aspects of the material under review, identify negative aspects constructively and indicate the improvements needed.”3

Constructive criticism does not, of course, include personal attacks. Regardless of the nature of the review, it is usually inappropriate for a reviewer to contact the author directly.
Peer Review
P 9/16
Competence
Reviewers need to disclose their level of competence to the editor. Reviewers do not necessarily have to be an expert with regard to every facet of a manuscript, but they need to inform the journal of their situation so that the editor can make an informed decision about selecting reviewers.
Peer Review
P 10/16
Impartiality and Integrity
Reviewers must review a manuscript and a grant proposal objectively and fairly. Reviewers are required to honestly evaluate their own ability to be impartial, in terms of whether they might have any personal, financial or ideological conflict of interest that could compromise an objective review, whether it is possible to make a fair decision, and the like. A review must be based solely on the manuscript or proposal’s merits, not on the reviewer’s personal interests.
Peer Review
P 11/16
Disclosure of Conflict of Interest
Reviewers must disclose conflicts of interest that could potentially compromise their ability to evaluate a manuscript honestly and objectively. In some circumstances, it may be appropriate for a reviewer to decline to review a manuscript because of a conflict of interest, such as a close friendship with the author. Reviewers should refer to the journal’s or funding agency’s policies on conflicts of interest in order to determine which specific information needs to be disclosed.
Peer Review
P 12/16
Timelines and Responsiveness
As best as possible, reviewers should follow the editor’s instructions for how the review should proceed. A reviewer’s obligations include completing the review in the time frame recommended by the editor. In general, if reviewers are unable to meet a journal’s deadline for the review, they should decline the opportunity to review the manuscript. They should do this as quickly as possible so that the editor can search for other prospective reviewers.
Peer Review
Potential Problems with Peer Reviews
P 13/16
There are some criticisms of the peer review process, which include:
  • Reviewers may have biases that they are unable to disregard when they read a manuscript or grant application. Such biases can include disagreements with methods used in a manuscript or grant, dislike for an author’s or applicant’s institution, dislike of the author or applicant, and competition with the author or applicant.
  • Reviewers are often selected from supporters of the dominant theory in the field, thus, a peer review might prevent controversial or innovative research from entering into the literature or from being used as the basis for a grant application.
  • Reviewers may not be forthcoming in admitting conflicts of interest that they have relating to a manuscript or grant application, due to their lack of awareness of the conflicts of interest.
  • Reviewers may not admit their lack of expertise in reviewing a manuscript or grant application.
  • The peer review process does not always detect errors, and, in some cases, may not detect willful misconduct such as fabrication or falsification of data.
  • The gender of peer reviewers may produce different review results. There are research findings that have shown that when an author is a woman, the manuscript has more of a chance to be adopted by non-male reviewers.
Even with flaws, the peer review process generally contributes to the improvement of the quality of the research presented in a manuscript or a grant application. Awareness of the problems inherent in the process of peer review, such as the potential for bias and plagiarism of information, often helps people to avoid getting into ethical problems.
Peer Review
P 14/16
Resolving Problems with a Review
If authors believe that a manuscript has been rejected unfairly, they can write to the journal editor and discuss the relevant concerns. There are appeals in the grant application process as well. For example, if someone considers that their work might have been plagiarized from a peer review, then the author or grant applicant could seek legal advice and perhaps contact the institution where the peer reviewer works. The institution should have an office that deals with the alleged misconduct. Contacting the granting agency or the journal might also be appropriate.
Peer Review
P 15/16
Improving the Peer Review Process
Several improvements for solving problems associated with peer review have been proposed and put into practice.

Double-Blind:
The “double-blind” review process, mentioned above as a method of peer review, was produced as one solution to the problems that could occur in the peer review process. When it works, it eliminates all potential biases in the peer review process which arise due to the knowledge of the author’s identity. Some point out that manuscripts submitted by highly acclaimed researchers of elite universities are likely to be graded higher than those submitted by lesser-known researchers of lesser-known institutions. Also, some make the point that even if the double-blind review process is used, speculation about authors’ identities can still take place depending on the field of research. Yet, by allowing reviewers to determine the significance of research topics, focus on the central issue and evaluate methods of research, the “double-blind” review process is superior relative to the “single-blind” review process. It is also believed to increase the level of confidentiality and fairness of the process.

Open Review:
Fiona Godlee, the editor of the British Medical Journal (BMJ), proposes that quality peer reviews may be achieved by making all processes open so that reviewers can know who the authors are and vice versa.4 Godlee suggests that accountability would be enhanced if authors and reviewers knew each other’s identities, because reviewers would be less inclined to offer unjustified arguments or to misappropriate data which may be a problem inherent in the anonymous peer review process. Also, an open review process could provide opportunities for authors to point out reviewers’ conflicts of interest, which might have been overlooked by editors. She concedes, however, that many reviewers might refuse to participate in such a system.
Peer Review
Conclusion
P 16/16
Reviewing the manuscripts and grant applications of others is a vital activity for research communities. Peer reviewers need to be aware of their responsibilities to editors, authors and, ultimately, readers. Maintaining the integrity of the peer review process is also essential for it to maintain its effectiveness.


Peer Review
References
This module is adapted for the APRIN research community from “Peer Review (RCR-Biomedical)” kindly offered by the CITI Program of BRANY (Biomedical Research Alliance of New York). Adaption was carried out by the APRIN supporting experts in accordance with the various pertinent laws and guidelines, whose names are listed elsewhere.
Please provide feedback on this material