Prologue

Many years ago when I was bemoaning my first manuscript rejection (of many), I complained to a colleague that the reviewers had gotten it all wrong. My colleague kindly told me that it was my fault. It was my job to write clearly enough so that the reviewers could easily understand what I was trying to say. I realize now how right he was. Saunders (2005)

Introduction

Search for diamonds

Journals are happy to accept high quality papers. However, acceptance rates of many A journals are around 10% for several reasons.

Wearing the reviewer’s hat, one task is to be critical to ensure high quality but at the same time to search for the gem in the paper and to cut this diamond.

Expert diamond cutters may be able to handle some errors in the diamond, but fatal flaws are impossible to conceal.

Editors and reviewers task is to search for the gem in the paper and to cut this diamond.

What makes up a rough diamond?

Recap

I believe that it is the unbeatable combination of a reasonably well applied theory answering interesting, novel questions in a well known thematic stream of work that leads to blue oceans. Straub (2009)

Move into intellectual territory that is unexploited and pose fascinating research questions.
What is new here? Where is the ‘hook’?

Link your topic to popular themes—join a major conversation

Theory is king—it is in the evaluation of the theoretical contribution that most reviewers become convinced, or not.

Good manuscripts

Pay attention to the following criteria (Zmud 1998; Saunders 2005).

Strong motivation: Infect the mind of your reader with one clear, sharp idea and be explicit about it, its significance and novelty.

Significant contribution: Work out your findings clearly and place them well in the context of ongoing research.

Appropriate research design: Ensure a convincing logic to your theoretical or analytical argument and the appropriateness of data collection and analysis.

Good presentation: Write in such a way that the reviewers can easily understand what you are trying to say.

Exercise

Reflect your read of hypothetical AoM micro submission “Responses to Transformational Leadership: Are Some Followers Immune?”.

Form small groups and judge it on the criteria outlined above.

My thesis criteria

Motivation, theory, method, results, contribution, and writing

When evaluating research as told in a paper or thesis, I pose following questions:

Introduction

Is there one clear, sharp idea that is significant and novel?
Is the research question (or research aim) clearly stated and justified?
Given the introduction, does the paper seem well structured, well grounded in theory, and to follow a structured approach?

Theory

Does the paper/thesis contain a well-developed and articulated theoretical framework?
Are the core concepts (technical terms) of the paper clearly defined?
Is the theoretical and analytic argument convincing?
Is extant literature appropriately reflected in the paper, or are critical references missing?

Method (for empirical papers)

Are the sample and variables appropriate?
Is the data collection and data analysis method appropriate?
Is internal and external validity sufficiently adressed?

Results (for empirical papers)

Are the results reported in an understandable way?
Are the results connected with the theory?

Contribution

Does the paper answer the research question?
Is there a value-add (what do we learn, what was not already known)?
Are limitations critically reflected?
Do the authors discuss the implications of the work for the scientific and practice community?

Writing

Is the writing clear?
Is the paper concise?
Is the structure easy to follow?

Peer-reviews

Basics

Scientific publications are peer-reviewed.

Before a paper is published, it is reviewed by experts in the field
Most often double-blind process: authors and reviewers do not know each others’ identity
The review process is managed by an editor, which typically is a successful senior scholar
The editor adequate reviewers and makes decisions about the publication of the manuscript (incl. desk-rejects)

Desk-reject

Some papers are rejected without even being sent to reviewers. Why?

Lack of fit to the journal
- The paper does not cite any papers in the journal
- The üaper is not in the domain or not general enough
- Lack of fit is not an evaluation of quality!
Lack of likelihood
- The paper does not have a good chance of successfully passing the review process
- Protection of scarce resources: the reviewers
Lack of formalities
- The paper does not adhere to the submission requirements
- The language ist not good enough

Process

Outcomes

Submitting a paper for review leads to one of three outcomes:

Rejection, revision (major or minor), or (conditional) accept.

Rejection: outlet will not consider this manuscript or any revision of it (final decision)
Request to revise and resubmit: a reworked manuscript may be accepted if submitted to the journal again for new evaluation, typically by the same editor and reviewers
- Major revision: and extensive changes required, still high risk of getting rejected
- Minor revision: comments, issues, and suggestion for improvements, lower risk
Acceptance or conditional acceptance: the paper can be published without further review (as is, or with minor changes)

These results are ordered from worst to best and by frequency of occurrence. Acceptance rates are low; only about 10% of papers submitted to most journals are accepted. Virtually all initial reviews (“first-round” reviews) require significant revisions or re-submission if they are not rejected out of hand. In fact, you should assume that (a) the paper will not be accepted in the first round and (b) extensive revisions will be required (Recker 2021).

Sarcasm with a touch of reality

Review

What a review is

“Something that will ruin your day.” Even if it is good.
The stamp of scientific quality.
Feedback from your peers—future directions?

What a review is not

An acceptance/rejection decision (only editors or program committees accept or reject, reviewers recommend)
A place for bias, prejudice, personal animosity (though it often appears to be so)

Reasons

There are several key reasons why you should review other manuscripts:

Serve your duty as someone who submits manuscripts to peer-reviewed outlets (2-3 reviews written per paper written)
Grow your own research and writing skills (“good reviewers write good papers”)
Build the discipline, move forward science
Expand your network (the editors know you and will yeek you out as experts in the future)

Reviewing

Process

My suggestions for doing a review:

Read the paper
Read the review form (dimensions to look at)
Read the paper
Review/research literature
Note areas for enhancement
Wait a few days
Read paper
Write review

Put yourself in the author’s shoes and offer constructive criticism.

Try to write reviews you would like to get! I like reviews most, that do not just tell me that something is inadequate, but guide me in fixing it.

You should also study the other reviews on the paper as well as the editors decision.

Additional advices

Be polite and conversational,
identify some strengths,
search for the gem (the ideas),
be concise, be consistent, and be on time.

There is no clear preferred strategy for organizing comments to authors. Some reviewers organize their comments according to the evaluation dimensions. Others address the points in the order in which they appear in the paper. Still others organize their comments by importance: the most critical points first, followed by relatively unimportant points. Choose the most appropriate approach and also consider the following:

Be polite and conversational: be “author friendly” in your tone, and use terms like “you” instead of “the authors”
Be concise: summarize your comments on a topic in a single point, cite page numbers when referring to specific sections
Identify some strengths: open your review with what you liked before focusing on your criticisms and concerns
Be consistent: align the message of the comments to the author, the comments to the editor and your judgement on the form
Non-English native authors: distinguish between the quality of the writing and the quality of the ideas that the writing conveys
Be on time: return your review on time so that the editor can guarantee the authors a quick turnaround

Exercise

Read one of the reviews on the Hypothetical AoM micro submission “Responses to Transformational Leadership: Are Some Followers Immune?” and outline whether it conforms to the points outlined hitherto.

Revising

Basics

Do not to let pride or arrogance get in your way Recker (2021)

If the paper is rejected: determine why and fix legitimate flaws
If you are asked to revise and resubmit, do it
Take all editorial and reviewer comments seriously
See them as a chance to improve

Process

Recker (2021) recommends to manage revisions as follows:

Put the reviews aside for a while
Read and understand the reviews
Tabulate reviews
Revise the paper
Write a response letter

Put the reviews aside for a while

All initial reviews are critical(you will very rarely accepted straightaway – it only happened to me once)
You will probably have some strong negative reactions to a review
After several days, they will look more manageable

Read and understand the reviews

You need to truly and fully understand each and every comment.
Read the reviews several times.
Develop a strategy

Tabulate reviews

Create a table for the editors and reviewers and copy and paste each “block” of review comments into a separate row
Have three columns, one for the comment, one for the response, and one for notes and discussion within the research team
Mark them as “quick fixes” (easy to handle) and “major pieces of work”
Indicate the type of revision required:
1. change the research,
2. change the presentation of research, or
3. a suggestion that cannot be handled given the research conducted

Revise the paper

Try to make your revisions developmental
Implement the changes that will have to be made (ranging from changing, inserting, or deleting paragraphs, figures, or tables of some sections, to writing a completely new manuscript)
Don’t be afraid to make big changes to a manuscript. A major revision should always look like one
Some suggest completely re-writing a paper from scratch for major revisions

Write a response letter

Response letters serve as a frame of reference for the review team when they are looking at revisions. It has usually been months since they read the original paper, and they want to be reminded and know what you changed and where they can find the changes in the paper. Since the response letter is a support document, it should be as brief as possible, but as concise as necessary.

Maintain a polite, professional, and constructive tone
Expresses your gratitude towards the editor and reviewers
Outline the strategy your pursued in your revision and summarize the main changes made
Write point-by-point responses to all comments made using the revision table (interpretation of the comment, revision alternatives if applicable, justification of chosen revision, and description of changes), for changes you have not incorporated as requested, provide a clear and compelling rationale

Response table

Many outlets recommend to format responses in a table (e.g., Techatassanasoontorn and Davison 2022)

Figure 3: Example for a point-by-point response(Techatassanasoontorn and Davison 2022)

Additional comments

Develop a revision strategy document, engage in an intellectual conversation, and produce concise responses.

Develop a revision strategy document: reflect on review comments, develop a strategy (understanding of the reviews and planned actions) and share this with the senior editor, requesting feedback.
Engage in an intellectual conversation: see the process as a way to learn and try to see our work from a different perspective
Produce concise responses: summarize the author’s understanding of the issues, how the author addresses them and indicate changes in the manuscript (page, paragraph, lines) for each response

Cover letter

Example revision cover letter:

Exercise

Image you would be the author of the hypothetical AoM micro submission “Responses to Transformational Leadership: Are Some Followers Immune?”.

Pick one point of one of the reviews and write a short, hypothetical response to the critique.

Q&A

Literature

Recker, Jan. 2021. Scientific Research in Information Systems: A Beginner’s Guide. Springer Nature.

Saunders, Carol. 2005. “Editor’s Comments: Looking for Diamond Cutters.” MIS Quarterly, iii–viii.

Straub, Detmar W. 2009. “Editor’s Comments: Why Top Journals Accept Your Paper.” MIS Quarterly, iii–x.

Techatassanasoontorn, Angsana A, and Robert M Davison. 2022. “Scholarly Conversation Through a Review Response Document.” Information Systems Journal.

Zmud, Bob. 1998. “A Personal Perspective on the State Journal Refereeing.” MIS Quarterly 22 (3): R45.