# Conventional guidelines for referee reports

## **How to write a good review (general conventional guidelines)**

<details>

<summary>Some key points</summary>

* Cite evidence and reference specific parts of the research when giving feedback.
* Justify your critiques and claims in a reasoning-transparent way, rather than merely ‘"passing judgment." Avoid comments like "this does not pass the smell test".
* Provide specific, actionable feedback to the author where possible.
* Try to restate the authors’ arguments, clearly presenting the most reasonable interpretation of what they have written. See [steelmanning](https://www.lesswrong.com/tag/steelmanning).
* Be collegial and encouraging, but also rigorous. Criticize and question specific parts of the research without suggesting criticism of the *researchers themselves.*

</details>

We're happy for you to use whichever process and structure you feel comfortable with when writing your evaluation content.

<details>

<summary>One possible structure</summary>

*Core*

1. *Briefly* summarize the work in context
2. Highlight positive aspects of the paper and its strengths and contributions, considered in the context of existing research.
3. *Most importantly:* Identify and assess the paper's most important and impactful claim(s). Are these supported by the evidence provided? Are the assumptions reasonable? Are the authors using appropriate methods?
4. Note major limitations and potential ways the work could be improved; where possible, reference methodological literature and discussion and work that models what you are suggesting.

*Optional/desirable*

* Offer suggestions for increasing the impact of the work, for incorporating the work into global priorities research and impact evaluations, and for supporting and enhancing future work.
* Discuss minor flaws and their potential revisions.
* Desirable: formal ['claim identification and assessment'](https://docs.google.com/document/d/1mBkAmCVomcUt0Ks7hsxShTsjAbx3WVtFfMCnasGQxns/edit#heading=h.ljcrdyqus3l8)

Please *don't* spend time copyediting the work. If you like, you can give a few specific suggestions and then suggest that the author look to make other changes along these lines.

</details>

{% hint style="info" %}
**Remember**: *The Unjournal* doesn’t “publish” and doesn’t “accept or reject.” So don’t give an *Accept*, *Revise-and-Resubmit'*, or *Reject*-type recommendation. We ask for quantitative metrics, written feedback, and expert discussion of the validity of the paper's main claims, methods, and assumptions.
{% endhint %}

## Writing referee reports: resources and benchmarks

**Economics**\
[How to Write an Effective Referee Report and Improve the Scientific Review Process (Berk et al, 2017)](https://pubs.aeaweb.org/doi/pdfplus/10.1257/jep.31.1.231)

Semi-relevant: [Econometric Society: Guidelines for referees](https://www.econometricsociety.org/publications/econometrica/guidelines-referees)

Report: [Improving Peer Review in Economics: Stocktaking and Proposal (Charness et al 2022)](https://evalresearch.weebly.com/uploads/1/3/3/4/133478410/improving_peer_review_in_economics_-_charness_et_al..pdf)

**Open Science**

[PLOS](https://plos.org/resource/how-to-write-a-peer-review/) (Conventional but open access; simple and brief)

[Peer Community In... Questionnaire ](https://peercommunityin.org/2022/05/20/questionnaire-for-reviewers/)(Open-science-aligned; perhaps less detail-oriented than we are aiming for)

[Open Reviewers Reviewer Guide ](https://zenodo.org/record/5484087)(Journal-independent “PREreview”; detailed; targets ECRs)

**General, other fields**

[The Wiley Online Library](https://authorservices.wiley.com/Reviewers/journal-reviewers/how-to-perform-a-peer-review/step-by-step-guide-to-reviewing-a-manuscript.html) (Conventional; general)

["Peer review in the life sciences (Fraser)"](https://fraserlab.com/peer_review/) (extensive resources; only some of this is applicable to economics and social science)

#### Other templates and tools

[Collaborative template: RRR assessment peer review](https://docs.google.com/document/d/1Oz6dHybbpVa2vgyl4bULHO7jS2DahtcQ8bknygFNYa8/edit#heading=h.gjdgxs)

\
[Introducing Structured PREreviews on PREreview.org](https://content.prereview.org/introducing-structured-prereviews-on-prereview-org/)

\
‘the 4 validities’ and [seaboat](https://www.seaboat.io/)\\