1 of 6

Pilot steps

See sections below.

Pilot: Building a founding committee

7 Feb 2023: We have an organized founding/management committee, as well as an advisory board (see Our team). We are focusing on pushing research through the evaluation pipeline, communicating this output, and making it useful. We have a working division of labor, e.g., among "managing editors," for specific papers. We are likely to expand our team after our pilot, conditional on further funding.

Progress: the team (continual update)

Our team

Key elements of plan

Put together founding committee, meetings, public posts, and feedback (done)

Build a "founding committee" of 5–8 experienced and enthusiastic EA-aligned or adjacent researchers at EA orgs, research academics, and practitioners (e.g., draw from speakers at recent EA Global meetings).
1. Create private Airtable with lists of names and organizations
2. Added element: List of supporter names for credibility, with little or no commitment
Host a meeting (and shared collaboration space/document), to come to a consensus on a set of practical principles. [26 May 2022: First meeting held, writing up shared notes.]
Post and present our consensus (coming out of this meeting) on key fora. After a brief followup period (~1 week), consider adjusting the above consensus plan in light of the feedback, repost, and move forward.

... Excerpts from successful ACX grant, , reiterated in followup FTX Future Fund (for further funding; unsuccessful).

How was this founding committee recruited?

The creation of an action plan can be seen in the Gdoc discussion "Procedure for choosing committee"
EA Forum question post: Soliciting lists and names

Three key relevant areas from which to draw candidates

DR: I think I need to draw people from a few relevant areas: 1. Academia, in relevant subject fields for The Unjournal: economics, quantitative social science, maybe more

2. Effective altruism, to assess the value and scope of the journal and the research

3. Open Science and academic reform, and applied metascience—people with practical ideas and knowledge

+ People with strong knowledge of the journal and bibliometric processes and systems

First: direct outreach to a list of pivotal, prominent people

Assemble a list of the most relevant and respected people, using more or less objective criteria and justification.
1. Ask to join founding committee.
2. Ask to join list of supporters.
Add people who have made past contributions.

28 May 2022: The above has mostly been done, at least in terms of people attending the first meeting. We probably need a more systematic approach to getting the list of supporters.

Second: public call for interest

Further posts on social media, academic websites and message boards, etc.

Pilot: Identifying key research

()

Test-case research for proof of concept

Identify a small set of papers or projects as representative first-cases; use to help test the system we are building in a concrete manner.

In doing the above, we are also collecting a longer list of key papers, projects, authors, topics, issues, etc.

Steps taken

Post on EA Forum (and other places) and present form (see view at bottom of this section) promoting our call for papers further, with bounty.

Rules for bounty HERE

2. Search for most-cited papers (within our domain) among EA-aligned researchers and organizations.

3. Dig into existing lists, reviews, and syllabi, such as:

GPI research agenda (includes many posed questions)
Open Philanthropy "questions that might affect our grantmaking" (needs updating? few academic links)
The EA Behavioral Science Newsletter
Syllabi: Pablo's list; Economics focus list; David Rhys-Bernard's syllabus (link to my commented/highlighted version)

Consider: "Mistakes" pages?

Givewell (mainly operational mistakes)

ACX/Scott Alexander

Not very relevant because focused on operational issues

5. Appeal directly to authors and research groups

6. Cross-promote with How to get involved

Pivot: direct focus on NBER working papers

"Direct evaluation" track

Pete Slattery: "Do a complete test run using a single paper and team…" Thus, we aim to identify a small set of papers (or projects), maybe 2–3, that seem like good test and example cases, and offer a bounty for projects we choose as test cases.
Note that much of the work identified here has already been peer-reviewed and "published." While we envision that The Unjournal may assess papers that are already published in traditional journals, these are probably not the best case for the PoC. Thus, we de-prioritize these for now.

Pilot: Setting up platforms

Set up the basic platforms for posting and administering reviews and evaluations and offering curated links and categorizations of papers and projects.

Progress reports

Update 7 Sep 2022, partial update 22 Dec 2022

We are setting up processes and forms in Kotahi
- Submissions form is pretty useable (but imperfect, e.g., we need to ask people to (click 'submit a URL instead' on page one)
Evaluations form: using a Gdoc for now, trying out Airtable, Qualtrics and other solutions, aiming to integrate it into Kotahi

See Mapping evaluation workflow for how projects will enter, be evaluated, and 'output'
We will outline specific requests for developers\
Sciety group set up with 'Hypothes.is feed'; working on processing first evaluations\

Submission, evaluation and management platform Kotahi: submit/eval/mgmt (may be phasing out?)

7 Feb 2023

Set up Kotahi page HERE
Configured it for submission and management
Mainly configured for evaluation but it needs bespoke configuration to be efficient and easy for evaluators, particular for the quantitative ratings and predictions. Thus we are using Google Docs (or cryptpads) for the pilot. Will configure Kotahi with further funds.

Sciety group (curated evaluations and research)

Evaluations are curated in our Sciety.org group, which integrates these with the publicly-hosted research.

7 Feb 2023: We are working on

The best ways to get evaluations from "submissions on Kotahi" into Sciety,
... with the curated link to the publicly-hosted papers (or projects) on a range of platforms, including NBER
Ways to get DOIs for each evaluation and author response
Ways to integrate evaluation details as 'collaborative annotations' (with hypothes.is) into the hosted papers

(We currently use a hypothes.is workaround to have this feed into Sciety so these show up as ‘evaluated pre-prints’ in their public database, gaining a DOI.

Notes, exploring the platform.

Setting up evaluation guidelines for pilot papers

7 Feb 2023 We have considered and put together:

See: Guidelines for evaluators

... including descriptive and quantitative (rating and prediction elements). With feedback from evaluators and others, we are continuing to build and improve these guidelines.

'Evaluators': Identifying and engaging

Status: 7 Feb 2023

Volunteer pool of 80+ reviewers (see Airtable), responding to How to get involved and other outreach
For our initial 3 focal pilot papers we have a total of 8 completed evaluations (2 of these are completed subject to some final checks.
For the remaining 7 pilot papers, we have roughly 6 agreed evaluators so far (we aim for 2-3 per paper)

Pilot: Setting up platforms

Set up the basic platforms for posting and administering reviews and evaluations and offering curated links and categorizations of papers and projects.

Progress reports

Update 7 Sep 2022, partial update 22 Dec 2022

We are setting up processes and forms in Kotahi
- Submissions form is pretty useable (but imperfect, e.g., we need to ask people to (click 'submit a URL instead' on page one)
Evaluations form: using a Gdoc for now, trying out Airtable, Qualtrics and other solutions, aiming to integrate it into Kotahi

See Mapping evaluation workflow for how projects will enter, be evaluated, and 'output'
We will outline specific requests for developers\
Sciety group set up with 'Hypothes.is feed'; working on processing first evaluations\

Submission, evaluation and management platform Kotahi: submit/eval/mgmt (may be phasing out?)

7 Feb 2023

Set up Kotahi page HERE
Configured it for submission and management
Mainly configured for evaluation but it needs bespoke configuration to be efficient and easy for evaluators, particular for the quantitative ratings and predictions. Thus we are using Google Docs (or cryptpads) for the pilot. Will configure Kotahi with further funds.

Sciety group (curated evaluations and research)

Evaluations are curated in our Sciety.org group, which integrates these with the publicly-hosted research.

7 Feb 2023: We are working on

The best ways to get evaluations from "submissions on Kotahi" into Sciety,
... with the curated link to the publicly-hosted papers (or projects) on a range of platforms, including NBER
Ways to get DOIs for each evaluation and author response
Ways to integrate evaluation details as 'collaborative annotations' (with hypothes.is) into the hosted papers

(We currently use a hypothes.is workaround to have this feed into Sciety so these show up as ‘evaluated pre-prints’ in their public database, gaining a DOI.

Notes, exploring the platform.

Pilot: Building a founding committee

Progress: the team (continual update)

Our team

Key elements of plan

Put together founding committee, meetings, public posts, and feedback (done)

Build a "founding committee" of 5–8 experienced and enthusiastic EA-aligned or adjacent researchers at EA orgs, research academics, and practitioners (e.g., draw from speakers at recent EA Global meetings).
1. Create private Airtable with lists of names and organizations
2. Added element: List of supporter names for credibility, with little or no commitment
Host a meeting (and shared collaboration space/document), to come to a consensus on a set of practical principles. [26 May 2022: First meeting held, writing up shared notes.]
Post and present our consensus (coming out of this meeting) on key fora. After a brief followup period (~1 week), consider adjusting the above consensus plan in light of the feedback, repost, and move forward.

... Excerpts from successful ACX grant, , reiterated in followup FTX Future Fund (for further funding; unsuccessful).

How was this founding committee recruited?

The creation of an action plan can be seen in the Gdoc discussion "Procedure for choosing committee"
EA Forum question post: Soliciting lists and names

Three key relevant areas from which to draw candidates

DR: I think I need to draw people from a few relevant areas: 1. Academia, in relevant subject fields for The Unjournal: economics, quantitative social science, maybe more

2. Effective altruism, to assess the value and scope of the journal and the research

3. Open Science and academic reform, and applied metascience—people with practical ideas and knowledge

+ People with strong knowledge of the journal and bibliometric processes and systems

First: direct outreach to a list of pivotal, prominent people

Assemble a list of the most relevant and respected people, using more or less objective criteria and justification.
1. Ask to join founding committee.
2. Ask to join list of supporters.
Add people who have made past contributions.

28 May 2022: The above has mostly been done, at least in terms of people attending the first meeting. We probably need a more systematic approach to getting the list of supporters.

Second: public call for interest

Further posts on social media, academic websites and message boards, etc.

Pilot: Identifying key research

()

Test-case research for proof of concept

Identify a small set of papers or projects as representative first-cases; use to help test the system we are building in a concrete manner.

In doing the above, we are also collecting a longer list of key papers, projects, authors, topics, issues, etc.

Steps taken

Post on EA Forum (and other places) and present form (see view at bottom of this section) promoting our call for papers further, with bounty.

Rules for bounty HERE

2. Search for most-cited papers (within our domain) among EA-aligned researchers and organizations.

3. Dig into existing lists, reviews, and syllabi, such as:

GPI research agenda (includes many posed questions)
Open Philanthropy "questions that might affect our grantmaking" (needs updating? few academic links)
The EA Behavioral Science Newsletter
Syllabi: Pablo's list; Economics focus list; David Rhys-Bernard's syllabus (link to my commented/highlighted version)

Consider: "Mistakes" pages?

Givewell (mainly operational mistakes)

ACX/Scott Alexander

Not very relevant because focused on operational issues

5. Appeal directly to authors and research groups

6. Cross-promote with How to get involved

Pivot: direct focus on NBER working papers

"Direct evaluation" track

Pete Slattery: "Do a complete test run using a single paper and team…" Thus, we aim to identify a small set of papers (or projects), maybe 2–3, that seem like good test and example cases, and offer a bounty for projects we choose as test cases.
Note that much of the work identified here has already been peer-reviewed and "published." While we envision that The Unjournal may assess papers that are already published in traditional journals, these are probably not the best case for the PoC. Thus, we de-prioritize these for now.

Pilot steps

Pilot: Building a founding committee

Progress: the team (continual update)

Key elements of plan

How was this founding committee recruited?

First: direct outreach to a list of pivotal, prominent people

Second: public call for interest

See also public Gdoc discussion of "the committee and consensus"

Pilot: Identifying key research

Test-case research for proof of concept

Steps taken

Pivot: direct focus on NBER working papers

Pilot: Setting up platforms

Submission, evaluation and management platform Kotahi: submit/eval/mgmt (may be phasing out?)

Sciety group (curated evaluations and research)

Setting up evaluation guidelines for pilot papers

'Evaluators': Identifying and engaging

Setting up evaluation guidelines for pilot papers

Pilot: Setting up platforms

Submission, evaluation and management platform Kotahi: submit/eval/mgmt (may be phasing out?)

Sciety group (curated evaluations and research)

Pilot steps

Pilot: Building a founding committee

Progress: the team (continual update)

Key elements of plan

How was this founding committee recruited?

First: direct outreach to a list of pivotal, prominent people

Second: public call for interest

See also public Gdoc discussion of "the committee and consensus"

'Evaluators': Identifying and engaging

Pilot: Identifying key research

Test-case research for proof of concept

Steps taken

Pivot: direct focus on NBER working papers