AI Reviewer (ResearchPilot™) – Rayyan Help Center

AI Reviewer is part of ResearchPilot™, Rayyan's suite of AI-powered research tools. It's an independent screening agent that joins your review as a second reviewer, assessing your review's Inclusion and Exclusion Criteria and making screening decisions in fully blind mode, without ever seeing how the other reviewers on your team have decided.

Activating AI Reviewer

AI Reviewer screens against the Inclusion and Exclusion Criteria defined for your review, so those criteria must be in place before it can run. It will not start on an empty criteria set.

Best Practice For the most reliable results, enter the exact criteria from your registered research protocol. AI Reviewer's assessments are only as precise as the criteria it's given, so the clearer and more specific each criterion is, the better its judgments will be.

To invite AI Reviewer to your review:

Go to the Overview tab of your review.
In the Review Members table, find AI Reviewer. It appears with Inactive status.
Click Activate. AI Reviewer is now a member of your review.

Once invited, choose what to run AI Reviewer on:

All references in Stage 1 screening (title and abstract), or
A subset of references, narrowed using Rayyan filters.

Once started, AI Reviewer can take up to 24 hours to assess the references and reach its decisions.

Note Only the review owner can invite and run AI Reviewer. Each run is limited to 2,000 references and a maximum of 20 combined Inclusion and Exclusion Criteria. If your review has more than 2,000 references, use Data Sampling to split your dataset into batches and run AI Reviewer on each one. See the section below.

Screening larger datasets with AI Reviewer

If your review has more than 2,000 references, use Data Sampling to split it into smaller batches, then run AI Reviewer separately on each one.

Go to the Screening stage and click Samples.
Under Choose sample Data Source, select All References, or Filtered if you want to sample from a subset, for example, references you've already narrowed down with a filter.
Select Randomized as the sampling method. This gives each batch a representative cross-section of your dataset.
Create samples of no more than 2,000 references each, until your full dataset is covered. Give each sample a clear name (for example, "Batch 1", "Batch 2").
In the Screening stage, open the Filters panel, enable the Sampled Data facet, and select the first batch.
Run AI Reviewer as usual (see Activating AI Reviewer above), and when prompted to choose what to run it on, select Filtered so it only evaluates the references in the selected batch. Repeat for each remaining batch until AI Reviewer has evaluated every reference.

Note Your Inclusion and Exclusion Criteria stay the same across every batch, only the reference selection changes. See How to Use Data Sampling in Rayyan for the full walkthrough on creating and managing samples.

How AI Reviewer works

Assessing each criterion

For every reference, AI Reviewer evaluates each criterion and marks it as one of three outcomes:

Met: the title and abstract provide clear evidence that the criterion is satisfied.
Not met: the reference provides clear evidence that the criterion is not satisfied.
Unclear: there is not enough evidence to decide, or the fulltext may hold more relevant information.

Reaching a decision

Once every criterion has been assessed, AI Reviewer applies these rules in order:

If any exclusion criterion is met, the article is excluded. Otherwise, move to the next rule.
If any inclusion criterion is not met, the article is excluded. Otherwise, move to the next rule.
If at least one inclusion criterion is met, the article is included. If none are, the article is marked maybe.

Every article receives a note laying out the reasoning behind each criterion assessment and explaining what triggered the final decision.

Viewing AI Reviewer's results

Like any reviewer, AI Reviewer operates under Blind Mode, so its decisions stay hidden from the rest of the team while screening is in progress. This keeps its judgments from influencing the human reviewers, and vice versa.

To see AI Reviewer's decisions and its per-criterion reasoning, the review owner turns off Blind Mode. Its results then appear alongside those of the other reviewers, where you can compare decisions and resolve any conflicts.

Best Practice AI Reviewer's decision cannot be changed. To resolve a conflict with AI Reviewer, use labels to record your team's final decision and the reasoning behind it, rather than trying to edit AI Reviewer's Include or Exclude choice.

Continue to: AI Analyzer (ResearchPilot™)