# Independent evaluations {% hint style="info" %} We welcome independent evaluations and offer prizes for the strongest submissions. Contact us at if you're interested. {% endhint %} {% hint style="info" %} [**Disambiguation**](#user-content-fn-1)[^1] {% endhint %} *Note on* [*other versions of this content*](#user-content-fn-2)[^2]*.* ### Initiative: ‘independent evaluations’ [The Unjournal](http://unjournal.org/) is seeking academics, researchers, and students to submit structured evaluations of the most impactful research [emerging in the social sciences](#user-content-fn-3)[^3]. Strong evaluations will be posted or linked on our [PubPub community](http://unjournal.pubpub.org/), offering readers a perspective on the implications, strengths, and limitations of the research. These evaluations can be submitted using[ this form](https://coda.io/form/Unjournal-Evaluation-form-academic-stream-Coda-updated-version_dGjfMZ1yXME) for academic-targeted research or[ this form](https://coda.io/form/Unjournal-evaluation-form-applied-stream_dkjUPyzvHoH) for [more applied work](#user-content-fn-4)[^4]; evaluators can publish their name or maintain anonymity; we also welcome collaborative evaluation work. We will facilitate, promote, and encourage these evaluations in several ways, described below. ### Who should do these evaluations? We are particularly looking for people with research training, experience, and expertise in quantitative social science and statistics including cost-benefit modeling and impact evaluation. This could include professors, other academic faculty, postdocs, researchers outside of academia, quantitative consultants and modelers, PhD students, and students aiming towards PhD-level work (pre-docs, research MSc students etc.) But anyone is welcome to give this a try — when in doubt, piease go for it. We are also happy to support collaborations and group evaluations. There is a good track record for this — see: “[What is a PREreview Live Review?](https://support.jmir.org/hc/en-us/articles/4408266275099-What-is-a-PREreview-Live-Review)”, ASAPBio’s[ Crowd preprint review](https://asapbio.org/crowd-preprint-review),[ I4replication.org](http://i4replication.org/) and[ repliCATS](https://replicats.research.unimelb.edu.au/) for examples in this vein. We may also host live events and/or facilitate asynchronous collaboration on evaluations ***Instructors/PhD, MRes, Predoc programs:*** We are also keen to work with students and professors to integrate ‘independent evaluation assignments’ (aka ‘learn to do peer reviews’) into research training. ### Why should you do an evaluation? Your work will support The Unjournal’s core mission — improving impactful research through journal-independent public evaluation. In addition, you’ll help research users (policymakers, funders, NGOs, fellow researchers) by providing high quality detailed evaluations that rate and discuss the strengths, limitations, and implications of research. Doing an independent evaluation can also help *you*. We aim to provide feedback to help you become a better researcher and reviewer. We’ll also give prizes for the strongest evaluations. Lastly, writing evaluations will help you build a portfolio with The Unjournal, making it more likely we will commission you for paid evaluation work in the future. ### Which research? We focus on rigorous, globally-impactful research in quantitative social science and policy-relevant research. (See [“What specific areas do we cover?”](https://globalimpact.gitbook.io/the-unjournal-project-and-communication-space/policies-projects-evaluation-workflow/considering-projects/what-specific-areas-do-we-cover) for details.) We’re especially eager to receive independent evaluations of: 1. Research we publicly prioritize: see our [public list of research](https://coda.io/d/Cross-Doc-Sync_d7VdSLeCrpi/Unjournal-Research-of-interest-database-public_suCwXMPL#_lunMtjpm) we've prioritized or evaluated. (Also...) 2. Research we previously evaluated (see [public list](https://coda.io/d/Cross-Doc-Sync_d7VdSLeCrpi/Unjournal-Research-of-interest-database-public_suCwXMPL#_lunMtjpm), as well as ) 3. Work that other people and organizations suggest as having high potential for impact/value of information (also see[ Evaluating Pivotal Questions](https://coda.io/d/_ddIEzDONWdb/_suamu)) *You can also suggest research yourself* [*here*](https://bit.ly/UJsuggest) *and then do an independent evaluation of it.* ### What sort of ‘evaluations’ and what formats? We’re looking for careful methodological/technical evaluations that focus on research credibility, impact, and usefulness. We want evaluators to *dig into the weeds*, particularly in areas where they have aptitude and expertise. *See our*[ *guidelines*](https://globalimpact.gitbook.io/the-unjournal-project-and-communication-space/policies-projects-evaluation-workflow/evaluation/guidelines-for-evaluators)*.* ***The Unjournal’s structured evaluation forms:** We encourage evaluators to do these using either:* 1. Our [Academic (main) stream form](https://coda.io/form/Unjournal-Evaluation-form-Academic-Coda-flexible-version_dGjfMZ1yXME): If you are evaluating research aimed at an academic journal *or* 2. Our [‘Applied stream’ form](https://coda.io/form/Unjournal-evaluation-form-applied-stream_dkjUPyzvHoH): If you are evaluating research that is probably *not* aimed at an academic journal. This may include somewhat less technical work, such as reports from policy organizations and think tanks, or impact assessments and cost-benefit analyses {% hint style="info" %} [See here for guidance on using these forms for independent evaluations](https://coda.io/d/_ddIEzDONWdb/_suMlX#_lullT) {% endhint %} ***Other public evaluation platforms:*** We are *also* open to engaging with evaluations done on existing public evaluation platforms such as[ PREreview.org](http://prereview.org/). *Evaluators*: If you prefer to use another platform, please let us know about your evaluation using one of the forms above. If you like, you can leave most of our fields blank, and provide a link to your evaluation on the other public platform. ***Academic (\~PhD) assignments and projects:*** We are also looking to build ties with research-intensive university programs; we can help you structure academic assignments and provide external reinforcement and feedback. *Professors, instructors, and PhD students:* please contact us (). ### How will The Unjournal engage? #### 1. Posting and signal-boosting We will encourage all these independent evaluations to be publicly hosted, and will share links to these. We will further promote the strongest independent evaluations, potentially [re-hosting them on our platforms](#user-content-fn-5)[^5] (such as [unjournal.pubpub.org](http://unjournal.pubpub.org/)) However, when we host or link these, we will keep them clearly separated and signposted as *distinct* from the commissioned evaluations; independent evaluations will not be considered *official*, and their ratings won’t be included in our [‘main data’](http://unjournal/) (see dashboard[ here](https://unjournal.github.io/unjournaldata/chapters/evaluation_data_analysis.html); see discussion[^6]). #### 2. Offering incentives ***Bounties:*** We offer prizes for the 'most valuable independent evaluations'. Strong submissions may receive prizes of $500 or more. [As a reference...](#user-content-fn-7)[^7] All evaluation submissions will be eligible for these prizes and “grandfathered in” to any prizes announced later. We will announce and promote the prize winners (unless they opt for anonymity). ***Evaluator pool:*** People who submit evaluations can elect to join our evaluator pool. We will consider and (time-permitting) *internally rate* these evaluations. People who do the strongest evaluations in our focal areas are likely to be commissioned as paid evaluators for The Unjournal. We’re also moving towards a two-tiered base [compensation for evaluations.](#user-content-fn-8)[^8] We will offer a higher rate to people who can demonstrate previous strong review/evaluation work. These independent evaluations will count towards this ‘portfolio’. #### 3. Providing materials, resources and guidance/feedback Our[ PubPub page](http://unjournal.pubpub.org/) provides examples of strong work, including the[ prize-winning evaluations](https://unjournal.pubpub.org/prize-winning-evaluations). We will curate guidelines and learning materials from relevant fields and from applied work and impact-evaluation. For a start, see[ "Conventional guidelines for referee reports" in our knowledge base.](https://globalimpact.gitbook.io/the-unjournal-project-and-communication-space/policies-projects-evaluation-workflow/evaluation/guidelines-for-evaluators/conventional-guidelines-for-referee-reports) [*We plan to build and curate more of this...*](#user-content-fn-9)[^9] #### 4. Partnering with academic institutions We are reaching out to PhD programs and pre-PhD research-focused programs. Some curricula already involve “mock referee report” assignments. We hope professors will encourage their students to do these through our platform. In return, we’ll offer the incentives and promotion mentioned above, as well as resources, guidance, and some further feedback **5. Fostering a positive environment for anonymous and signed evaluations** We want to preserve a positive and productive environment. This is particularly important because we will be accepting *anonymous* content. We will take steps to ensure that the system is not abused. If the evaluations have an excessively negative tone, have content that could be perceived as personal attacks, or have clearly spurious criticism, we will ask the evaluators to revise this, or we may decide not to post or link it. ### How does this benefit The Unjournal and our mission? 1. **Crowdsourced feedback can add value in itself;** encouraging this can enable some public evaluation and discussion of work that The Unjournal doesn’t have the bandwidth to cover 2. **Improving our evaluator pool and evaluation standards in general.** 1. Students and ECRs can practice and (if possible) get feedback on independent evaluations 2. They can *demonstrate* their ability this publicly, enabling us to recruit and commission the strongest evaluators 3. **Examples will help us build guidelines, resources**, and insights into ‘what makes an evaluation useful’. 4. This provides us opportunities to **engage with academia,** especially in Ph.D programs and research-focused instruction. ### About *The Unjournal* (unfold) [*The Unjournal*](http://unjournal.org/) commissions public evaluations of impactful research in quantitative social sciences fields. We are an alternative and a supplement to traditional academic peer-reviewed journals – separating evaluation from journals unlocks a [range of benefits](https://globalimpact.gitbook.io/the-unjournal-project-and-communication-space/benefits-and-features). We ask expert evaluators to write detailed, constructive, critical reports. We also solicit a set of structured ratings focused on research credibility, methodology, careful and calibrated presentation of evidence, reasoning transparency, replicability, relevance to global priorities, and usefulness for practitioners (including funders, project directors, and policymakers who rely on this research). While we have mainly targeted impactful research from *academia*, our [‘applied stream’](https://docs.google.com/document/d/1RwkmGJtaOcryK-tJs3mrs6DIAj411YRnsauaj6EE6WY/edit) covers impactful work that uses formal quantitative methods but is *not* aimed at academic journals. As of January 2026, we've commissioned over 100 evaluations of 53 papers, and published these evaluation packages [on our PubPub community](https://unjournal.pubpub.org/), linked to academic search engines and bibliometrics. See our [interactive dashboard](https://unjournal.shinyapps.io/uj-dashboard) for current statistics. [^1]: The Unjournal focuses on *commissioning* expert evaluations, guided by an ‘evaluation manager’ and compensating people for their work. (See the outline of our main process[ here](https://globalimpact.gitbook.io/the-unjournal-project-and-communication-space/policies-projects-evaluation-workflow)). We plan to continue to focus on that mode. Below we sketch an *additional* parallel but separate approach. [^2]: We are maintaining a [Google Docs version here](https://docs.google.com/document/d/1Cj6LuHpKYN6SR77t5YZYte1lqu_wGSu3zMB2dcEtmJw/edit?usp=sharing) for comments and suggestions; that version was published as a linkpost [on the EA Forum here](https://forum.effectivealtruism.org/posts/NsmpbxSXAGefpPqS2/unjournal-wants-you-to-do-an-evaluation-pilot-independent-1). The version on our Coda page should now link here, to the Gitbook, instead. [^3]: This includes economics, policy, and impact modeling. See ["what specific areas do we cover"](https://globalimpact.gitbook.io/the-unjournal-project-and-communication-space/policies-projects-evaluation-workflow/considering-projects/what-specific-areas-do-we-cover) for more specifics on our scope. [^4]: Academic: Any work that you think is being aimed at an academic peer reviewed journal.\\ Applied: Unlikely to be submitted to an academic journal, aimed at a different audience, written in a different style. See [UJ Applied & policy stream ](https://docs.google.com/document/d/1RwkmGJtaOcryK-tJs3mrs6DIAj411YRnsauaj6EE6WY/edit#heading=h.o7xzpvh0h0b2). [^5]: We will also enable these to be posted *anonymously*, if the evaluators prefer. Evaluators can also choose to be ‘anonymous to the world but recognized by The Unjournal management’ for incentives and future commissions. [^6]: We aim to make Unjournal evaluations valuable for research-users and for building research careers, complementing and ultimately supplanting traditional ‘journal tier’ metrics. To this end, we generally recruit managers who select and invite evaluators with a balance of expertise in key areas that need evaluation, working to avoid biases and conflicts of interest. In contrast, evaluators acting on their own initiative might select papers they have a particular personal connection to, or a grievance against. In principle, an evaluation system based on voluntary/independent evaluations and reputation management systems could prove highly credible; e.g., Stackexchange seems to have achieved this in some adjacent areas. In the \~academic research context initiatives such as ResearchHub.com and Cooper Smout’s [“WISDOM”](https://github.com/openheartmind/WISDOM) lean in this direction. We will reconsider this in the future, as we learn from experience with independent evaluations. \\ [^7]: We previously awarded annual prizes between $250 and $2500 for strong research and evaluation work – [see here](https://docs.google.com/document/d/1zrw-D2Idmb0Yhgf8YelhEsOwYKkOyFfnARX-wu_Ch8s/edit#heading=h.ujoq1hbazath) [^8]: These base payments are in addition to one-off evaluator incentives and [evaluator prizes](https://globalimpact.gitbook.io/the-unjournal-project-and-communication-space/readme-1/latest-updates/impactful-research-prize-winners). [^9]: We aim at a balance of procedure, ethics, and etiquette, writing and communication, frameworks for considering research impact, the value-of-information, CBA etc, and substantive methodological considerations. While the latter are often field and context specific, some issues are relevant across a wide range of quantitative social science work and policy-relevant quantitative work (e.g., [The Difference Between “Significant” and “Not Significant” is not Itself Statistically Significant](http://www.stat.columbia.edu/~gelman/research/published/signif4.pdf); many posts from[ Data Colada](https://datacolada.org/toc); discussions of equivalence testing.) \\