How to Use Ratings and Relevance Ranking in Rayyan – Rayyan Help Center

Relevance Ranking is Rayyan’s AI-powered feature that analyzes your screening decisions and predicts how likely each undecided reference is to be included in your review. As you make more decisions, Rayyan learns from your patterns and assigns a rating to the remaining undecided references — helping you prioritize which articles to screen first. Ratings are available on all plans.

How the classifier works

Rayyan uses a Support Vector Machine (SVM) classifier trained on features extracted from each reference’s title and abstract — including unigrams (single words), bigrams (pairs of words), and MeSH Terms (Medical Subject Headings). These features are extracted after stopwords are removed and remaining terms are stemmed.

As you and your team make Include and Exclude decisions, Rayyan learns from those patterns. It calculates a confidence score for each unscreened reference based on how similar it is to your previous decisions, then translates that score into one of five rating levels.

Note Rayyan’s classifier was validated on 15 systematic review datasets using 2-fold cross-validation (repeated 10 times). It achieved an AUC of 0.87 ± 0.09 and a WSS@95 of 0.49 ± 0.18 — meaning reviewers could skip nearly half their remaining references while maintaining 95% recall. Read the full JAMIA study.

Minimum requirements to compute ratings

Ratings will not be calculated until you have made at least 50 screening decisions, with a minimum of 5 Include and 5 Exclude decisions. If this threshold has not been met, the Compute Ratings button will have no effect.

Common issue Seeing no ratings after running Compute Ratings? Check that you have at least 50 total decisions with at least 5 Include and 5 Exclude. Reviews early in the screening process or with very few exclusions will not produce predictions yet.

How to compute ratings

Go to the Screening tab
Click Compute Ratings in the toolbar
Rayyan processes your decisions and assigns a rating to each undecided reference

After the initial computation, you can recompute ratings as you continue screening. Rayyan can re-evaluate its model after every 10 minutes of additional training data received. If it detects that new decisions could improve its predictions, it retrains the classifier and re-scores the remaining undecided references. Recomputing does not affect any existing decisions.

The model stops retraining when all references have been decided or when it has reached optimal performance and cannot be further improved.

Best practice Recompute ratings after every major batch of decisions — roughly every 50–100 new decisions. The more balanced your Include/Exclude mix, the better the predictions will be.

How to read ratings

Each undecided reference is assigned one of five rating levels:

Rating	Icon	What it means
Most likely to include	👍👍	High confidence this reference meets your inclusion criteria
Likely to include	👍	Moderate confidence this reference should be included
No recommendation	❓	Not enough data to make a confident prediction — more decisions needed
Likely to exclude	👎	Moderate confidence this reference should be excluded
Most likely to exclude	👎👎	High confidence this reference does not meet your inclusion criteria

Common issue Seeing many “No recommendation” ratings? This means Rayyan doesn’t yet have enough data to make confident predictions — either the minimum threshold was just barely met, or your Include/Exclude mix is heavily skewed toward one decision. Recompute after making another balanced batch of decisions.

How to sort and filter by ratings

Once ratings are computed, you can sort references by clicking the sorting arrow next to the Filter by Inclusion dropdown. The sort uses Rayyan’s precise internal confidence score — more granular than the five displayed levels. You can also filter by rating level: open the Filters panel, find the Ratings facet, and select one or more levels to narrow your list — for example, showing only “Most likely to include” references to focus your screening on highest-priority articles.

Note Sorting by ratings is available on all plans. Filtering by rating level using the Ratings facet is available on Advanced and Institutional plans.

Ratings in collaborative reviews

Ratings are computed from all team decisions, even with Blind Mode on. Conflicting decisions — where two reviewers made opposing Include/Exclude choices on the same reference — are excluded from rating computation to avoid skewing predictions. Ratings are suggestions only and are never recorded as decisions or shared between reviewers.

Continue to: How to Use Review Chat in Rayyan