Rayyan Prediction Classifier

Rayyan's built-in classifier uses machine learning to assist with screening by learning from your inclusion/exclusion decisions. Here’s a breakdown of how it works and the research behind it.

🔍 Classifier Overview

Rayyan uses a Support Vector Machine (SVM) classifier trained on key features extracted from each citation’s title and abstract, including:

• 🧩 Unigrams (single words)

• 🧩 Bigrams (pairs of words)

• 🧬 MeSH Terms (Medical Subject Headings)

These features are extracted after stopwords are removed and remaining terms are stemmed.

Although we published a paper on a different classifier (Springer article), we opted not to use that model in Rayyan. Some of the features in that study, such as co-citations, are difficult to incorporate in a real-time production system.

📌 Note: We're currently working on a new publication focused entirely on Rayyan and its unique approach.

⚙️ How It Learns

As you (and your team) make inclusion/exclusion decisions, Rayyan's classifier starts to learn from your patterns. Once you’ve made at least:

• ✅ 50 screening decisions

  • With minimum 5 "Include" and 5 "Exclude"

…the system trains the model to classify the remaining undecided articles.

It then calculates a confidence score for each unscreened article — based on how similar it is to your previous decisions — and translates it into a thumbs up (Include) or thumbs down (Exclude) rating.

🔁 Continuous Improvement

As you keep screening, Rayyan automatically re-evaluates its model. If it detects that new labeled examples could improve its predictions, it retrains the classifier and re-scores the remaining undecided citations. This process continues until:

• 🗃️ All citations are labeled
• 🧠 The model reaches optimal performance and can't be further improved

📊 Validated Performance

Rayyan’s classifier was tested using the same features in a previous study:
➡️ Read the full JAMIA study

Study Highlights:

• 📚 15 systematic review datasets

• 🧪 2-fold cross-validation (repeated 10 times, with 50% training / 50% testing)

• 🧮 Metrics used:

  • AUC (Area Under the Curve): 0.87 ± 0.09

  • WSS@95 (Work Saved over Random Sampling): 0.49 ± 0.18

WSS@95 tells you how many citations reviewers didn’t have to screen thanks to the classifier, while still maintaining 95% recall.

📘 Want to Dive Deeper?

You can check out more technical details in our original publication:
➡️ Machine Learning for Systematic Reviews - Springer Article

While that study used Random Forest and extra features (like co-citations), those methods aren’t feasible for real-time use within Rayyan — which is why we rely on our current, streamlined SVM-based model.

💬 Still Need Help?

We’re here for you. Submit a support ticket and we’ll assist you personally.

And don’t forget to follow us on Twitter @rayyanapp for updates, tips, and tricks!