Review & revision

How to write and deal with reviews?

Andy Weeger

Neu-Ulm University of Applied Sciences

August 21, 2024

Opening remarks

Motivation

Many years ago when I was bemoaning my first manuscript rejection (of many), I complained to a colleague that the reviewers had gotten it all wrong. My colleague kindly told me that it was my fault. It was my job to write clearly enough so that the reviewers could easily understand what I was trying to say. I realize now how right he was. Saunders (2005)

Searching for diamonds

Journals are happy to accept high quality papers. However, acceptance rates of many A journals are around 10% for several reasons.

Wearing the reviewer’s hat, one task is to be critical to ensure high quality but at the same time to search for the gem in the paper and to cut this diamond.

Expert diamond cutters may be able to handle some errors in the diamond, but fatal flaws are impossible to conceal.

Editors and reviewers task is to search for the gem in the paper and to cut this diamond.

What makes up a rough diamond?

Recap

I believe that it is the unbeatable combination of a reasonably well applied theory answering interesting, novel questions in a well known thematic stream of work that leads to blue oceans. Straub (2009)

Move into intellectual territory that is unexploited and pose fascinating research questions.
What is new here? Where is the ‘hook’?

Link your topic to popular themes—join a major conversation

Theory is king—it is in the evaluation of the theoretical contribution that most reviewers become convinced, or not.

Good manuscripts

Pay attention to the following criteria (Zmud 1998; Saunders 2005).

Strong motivation: Infect the mind of your reader with one clear, sharp idea and be explicit about it, its significance and novelty.

Significant contribution: Work out your findings clearly and place them well in the context of ongoing research.

Appropriate research design: Ensure a convincing logic to your theoretical or analytical argument and the appropriateness of data collection and analysis.

Good presentation: Write in such a way that the reviewers can easily understand what you are trying to say.

My thesis criteria

Motivation, theory, method, results, contribution, and writing

When evaluating research as told in a paper or thesis, I pose following questions:

Introduction

• Is there one clear, sharp idea that is significant and novel?
• Is the research question (or research aim) clearly stated and justified?
• Given the introduction, does the paper seem well structured, well grounded in theory, and to follow a structured approach?

Theory

• Does the paper/thesis contain a well-developed and articulated theoretical framework?
• Are the core concepts (technical terms) of the paper clearly defined?
• Is the theoretical and analytic argument convincing?
• Is extant literature appropriately reflected in the paper, or are critical references missing?

Method (for empirical papers)

• Are the sample and variables appropriate?
• Is the data collection and data analysis method appropriate?
• Is internal and external validity sufficiently adressed?

Results (for empirical papers)

• Are the results reported in an understandable way?
• Are the results connected with the theory?

Contribution

• Does the paper answer the research question?
• Is there a value-add (what do we learn, what was not already known)?
• Are limitations critically reflected?
• Do the authors discuss the implications of the work for the scientific and practice community?

Writing

• Is the writing clear?
• Is the paper concise?
• Is the structure easy to follow?

Exercise

How does the hypothetical AoM micro submission “Responses to Transformational Leadership: Are Some Followers Immune?” comply with the criteria of good research and good writing discussed so far?

What recommendations would you give the authors?

Peer-reviews

Basics

Scientific publications are peer-reviewed.

• Before a paper is published, it is reviewed by experts in the field
• Most often double-blind process: authors and reviewers do not know each others’ identity
• The review process is managed by an editor, which typically is a successful senior scholar
• The editor adequate reviewers and makes decisions about the publication of the manuscript (incl. desk-rejects)

Desk-reject

Some papers are rejected without even being sent to reviewers. Why?

• Lack of fit to the journal
  • The paper does not cite any papers in the journal
  • The üaper is not in the domain or not general enough
  • Lack of fit is not an evaluation of quality!
• Lack of likelihood
  • The paper does not have a good chance of successfully passing the review process
  • Protection of scarce resources: the reviewers
• Lack of formalities
  • The paper does not adhere to the submission requirements
  • The language ist not good enough

Process

Figure 1: Publication process in peer-reviewed outlets

Outcomes

Submitting a paper for review leads to one of three outcomes:

Rejection, revision (major or minor), or (conditional) accept.

• Rejection: outlet will not consider this manuscript or any revision of it (final decision)
• Request to revise and resubmit: a reworked manuscript may be accepted if submitted to the journal again for new evaluation, typically by the same editor and reviewers
  • Major revision: and extensive changes required, still high risk of getting rejected
  • Minor revision: comments, issues, and suggestion for improvements, lower risk
• Acceptance or conditional acceptance: the paper can be published without further review (as is, or with minor changes)

These results are ordered from worst to best and by frequency of occurrence. Acceptance rates are low; only about 10% of papers submitted to most journals are accepted. Virtually all initial reviews (“first-round” reviews) require significant revisions or re-submission if they are not rejected out of hand. In fact, you should assume that (a) the paper will not be accepted in the first round and (b) extensive revisions will be required (Recker 2021).

Sarcasm with a touch of reality

Figure 2: Source: Science and Ink

Review

What a review is

• “Something that will ruin your day.” Even if it is good.
• The stamp of scientific quality.
• Feedback from your peers—future directions?

What a review is not

• An acceptance/rejection decision (only editors or program committees accept or reject, reviewers recommend)
• A place for bias, prejudice, personal animosity (though it often appears to be so)

Reasons

There are several key reasons why you should review other manuscripts:

• Serve your duty as someone who submits manuscripts to peer-reviewed outlets (2-3 reviews written per paper written)
• Grow your own research and writing skills (“good reviewers write good papers”)
• Build the discipline, move forward science
• Expand your network (the editors know you and will yeek you out as experts in the future)

Reviewing

Process

My suggestions for doing a review:

• Read the paper
• Read the review form (dimensions to look at)
• Read the paper
• Review/research literature
• Note areas for enhancement
• Wait a few days
• Read paper
• Write review

Put yourself in the author’s shoes and offer constructive criticism.

Try to write reviews you would like to get! I like reviews most, that do not just tell me that something is inadequate, but guide me in fixing it.

You should also study the other reviews on the paper as well as the editors decision.

Additional advices

Be polite and conversational,
identify some strengths,
search for the gem (the ideas),
be concise, be consistent, and be on time.

There is no clear preferred strategy for organizing comments to authors. Some reviewers organize their comments according to the evaluation dimensions. Others address the points in the order in which they appear in the paper. Still others organize their comments by importance: the most critical points first, followed by relatively unimportant points. Choose the most appropriate approach and also consider the following:

• Be polite and conversational: be “author friendly” in your tone, and use terms like “you” instead of “the authors”
• Be concise: summarize your comments on a topic in a single point, cite page numbers when referring to specific sections
• Identify some strengths: open your review with what you liked before focusing on your criticisms and concerns
• Be consistent: align the message of the comments to the author, the comments to the editor and your judgement on the form
• Non-English native authors: distinguish between the quality of the writing and the quality of the ideas that the writing conveys
• Be on time: return your review on time so that the editor can guarantee the authors a quick turnaround

Thesis reviews?

I advise you to form peer-groups and regularly submit parts of your work for review.

Exercise

Reflect the reviews on the Hypothetical AoM micro submission “Responses to Transformational Leadership: Are Some Followers Immune?” — do they conform to the points outlined hitherto?

Imagine you are part of the author team, which review would have been most helpful? Why?

Revising

Basics

Do not to let pride or arrogance get in your way Recker (2021)

• If the paper is rejected: determine why and fix legitimate flaws
• If you are asked to revise and resubmit, do it
  Take all editorial and reviewer comments seriously
  See them as a chance to improve

Process

Recker (2021) recommends to manage revisions as follows:

• Put the reviews aside for a while
• Read and understand the reviews
• Tabulate reviews
• Revise the paper
• Write a response letter

Put the reviews aside for a while

• All initial reviews are critical(you will very rarely accepted straightaway – it only happened to me once)
• You will probably have some strong negative reactions to a review
• After several days, they will look more manageable

Read and understand the reviews

• You need to truly and fully understand each and every comment.
• Read the reviews several times.
• Develop a strategy

Tabulate reviews

• Create a table for the editors and reviewers and copy and paste each “block” of review comments into a separate row
• Have three columns, one for the comment, one for the response, and one for notes and discussion within the research team
• Mark them as “quick fixes” (easy to handle) and “major pieces of work”
• Indicate the type of revision required:
  1. change the research,
  2. change the presentation of research, or
  3. a suggestion that cannot be handled given the research conducted

Revise the paper

• Try to make your revisions developmental
• Implement the changes that will have to be made (ranging from changing, inserting, or deleting paragraphs, figures, or tables of some sections, to writing a completely new manuscript)
• Don’t be afraid to make big changes to a manuscript. A major revision should always look like one
• Some suggest completely re-writing a paper from scratch for major revisions

Write a response letter

Response letters serve as a frame of reference for the review team when they are looking at revisions. It has usually been months since they read the original paper, and they want to be reminded and know what you changed and where they can find the changes in the paper. Since the response letter is a support document, it should be as brief as possible, but as concise as necessary.

• Maintain a polite, professional, and constructive tone
• Expresses your gratitude towards the editor and reviewers
• Outline the strategy your pursued in your revision and summarize the main changes made
• Write point-by-point responses to all comments made using the revision table (interpretation of the comment, revision alternatives if applicable, justification of chosen revision, and description of changes), for changes you have not incorporated as requested, provide a clear and compelling rationale

Response table

Many outlets recommend to format responses in a table (e.g., Techatassanasoontorn and Davison 2022)

Figure 3: Example for a point-by-point response(Techatassanasoontorn and Davison 2022)

Additional comments

Develop a revision strategy document, engage in an intellectual conversation, and produce concise responses.

• Develop a revision strategy document: reflect on review comments, develop a strategy (understanding of the reviews and planned actions) and share this with the senior editor, requesting feedback.
• Engage in an intellectual conversation: see the process as a way to learn and try to see our work from a different perspective
• Produce concise responses: summarize the author’s understanding of the issues, how the author addresses them and indicate changes in the manuscript (page, paragraph, lines) for each response

Cover letter

Example revision cover letter:

Figure 4: Example revision cover letter (Recker 2021)

Q&A

Literature

Recker, Jan. 2021. Scientific Research in Information Systems: A Beginner’s Guide. Springer Nature.

Saunders, Carol. 2005. “Editor’s Comments: Looking for Diamond Cutters.” MIS Quarterly, iii–viii.

Straub, Detmar W. 2009. “Editor’s Comments: Why Top Journals Accept Your Paper.” MIS Quarterly, iii–x.

Techatassanasoontorn, Angsana A, and Robert M Davison. 2022. “Scholarly Conversation Through a Review Response Document.” Information Systems Journal.

Zmud, Bob. 1998. “A Personal Perspective on the State Journal Refereeing.” MIS Quarterly 22 (3): R45.