Opening Remarks
Give and take
A short lecture on reviewing well, then you review, and are reviewed.
Everything you have built so far, topic, theory type, research question, has been tested only against your own team’s judgement and our feedback; this unit adds the stranger’s reading neither provides by pairing you with a team that does not share your assumptions, working only from what you actually wrote, to check whether your framing holds together.
Learning Outcomes
By the end of this unit, you will be able to:
- Write a structured, constructive review against an explicit rubric, citing specific claims rather than general impressions.
- Distinguish what a peer review is, feedback aimed at improvement, a check on your reasoning, from what it is not: the final call, or a place for personal point-scoring.
- Respond to a review of your own framing, incorporating each point or explaining, in writing, why you chose not to.
Peer Review Rationale
Basics
Scientific publications are peer-reviewed.
- Before a paper is published, it is reviewed by experts in the field.
- Most often a double-blind process: authors and reviewers do not know each other’s identity.
- The process is managed by an editor, an experienced scholar who selects reviewers and makes the publication decision.
Some papers are rejected without even being sent to reviewers. Why?
- Lack of fit: the paper is not in the domain, or not general enough for the outlet. This is not an evaluation of quality.
- Lack of likelihood: the paper does not have a good chance of successfully passing review, and reviewer time is a scarce resource worth protecting.
- Lack of formalities: the paper does not meet submission requirements, or the writing is not yet good enough to review productively.
Outcomes
Submitting work for review leads to one of three outcomes:
- Rejection: the outlet will not consider this manuscript, or any revision of it.
- Revise and resubmit: a reworked version may be accepted if resubmitted, typically to the same editor and reviewers. Major revision, extensive changes required, still high risk. Minor revision, comments and suggestions, lower risk.
- Acceptance or conditional acceptance: the work can be published as is, or with minor changes.
Virtually all first-round reviews require significant revision, if they are not rejected outright. Assume the same for your own work: your first draft of any deliverable will not be accepted as is, and extensive revisions are the normal case, not a sign something went wrong (Recker, 2021).
Review Scope
What a review is
- Feedback from your peers, aimed at real improvement.
- The stamp of methodological quality, someone besides you checked the chain of reasoning.
What a review is not
- The final call: reviewers recommend; the editor decides.
- A place for bias, prejudice, or personal point-scoring, though it can feel that way if done badly.
Reviewer Motivation
- Duty: you will receive reviews from others; you owe the same effort back.
- Skill-building: reviewing critically sharpens what you notice in your own framing. Good reviewers write good frames.
- The discipline moves forward: peer review is how a field’s collective standard stays higher than any one person’s.
Reviewing Well
Vague vs. Specific
Vague
“This doesn’t work. Rethink the question.”
Specific
“Your knowledge gap is about measurement inconsistency, but the question asks about causal mechanisms instead. State which one you mean, or explain how the mechanism produces the inconsistency.”
Both comments respond to the same flaw: a research question that does not follow from the knowledge gap the author stated. The vague version names a feeling, doesn’t work, without pointing at what to fix; it forces the author to guess. The specific version names the exact mismatch, quotes the two claims that don’t line up, and gives the author two concrete paths forward. The advice on the next slide operationalizes this difference: aim every comment you write at the specific version’s level of detail.
Process
- Read the framing.
- Read the review rubric (what to look for).
- Read the framing again.
- Note areas for enhancement.
- Wait a few days, if you can.
- Read once more, then write.
Put yourself in the author’s shoes, and offer constructive criticism.
Write reviews you would like to receive: reviews that point toward how to fix what is inadequate, beyond simply flagging it.
Additional Advice
- Be polite and conversational: write “you,” not “the authors.”
- Search for the gem: find the strongest version of what the team is trying to argue, and help them get there, beyond simply flagging its flaws.
- Be concise: one point per comment, reference specific lines or slides.
- Identify strengths first: open with what genuinely works before turning to concerns.
- Be consistent: your written comments and your overall verdict should tell the same story.
- Be on time: the other team is waiting on you the same way you will wait on them next time.
The Rubric
Checklist
This checklist replaces a vague “is this good” with four specific, checkable things.
- Is the chain unbroken? Domain → phenomenon → knowledge gap → research question. Can you follow it without the authors in the room to explain a jump?
- Which failure type, if any, is present? Monologuing, so-what, solving-the-world, multitude, or one of the five inappropriate-question categories from Good Research Questions.
- Which Gregor type does the question promise? Analysis, explanation, prediction, explanation and prediction, or design and action, and does the rest of the framing actually aim at that type consistently?
- Which theoretical lenses are doing real work, and which are decoration? A lens that is named but never used to generate a claim or a boundary condition is not yet earning its place.
This rubric is built directly from the last four units and introduces no new criteria. It operationalizes what you have already learned into something you can apply to someone else’s work in twenty minutes. If a criterion here feels unfamiliar, that is a signal to revisit the corresponding unit before you review, not to skip the criterion.
Common Failure Modes
- Halo effect: one strong element, a clean literature summary, a polished slide, makes you read the rest of the framing more favorably than it deserves.
- Confirmation bias: you review to confirm your first impression instead of testing whether the chain still holds up on a second, slower read.
- Anchoring on your own project: you end up flagging differences from what your own team would have done, rather than checking strictly against the rubric.
- Politeness inflation: you soften every point until the written review reads more generous than your private assessment, and the author cannot tell which point actually mattered.
Each failure mode pulls a review away from the rubric and toward the reviewer’s own state of mind. The halo effect and confirmation bias are documented across evaluation settings generally; naming them here is a reminder to check each rubric point independently on a second, slower pass rather than trusting a single overall impression formed on the first read. Anchoring on your own project and politeness inflation are specific to peer-review settings where reviewer and author are close in status and stake: guard against both by writing the review against the rubric, point by point, rather than against a running sense of whether the framing feels fine.
Homework
Revise your Problem Framing Brief in light of the review you received.
For each point raised in your review, either incorporate the change, or write one sentence explaining why you chose not to (to foster your own reflection).