Rayyan's built-in classifier uses machine learning to assist with screening by learning from your inclusion/exclusion decisions. Here’s a breakdown of how it works and the research behind it.
🔍 Classifier Overview
Rayyan uses a Support Vector Machine (SVM) classifier trained on key features extracted from each citation’s title and abstract, including:
-
🧩 Unigrams (single words)
-
🧩 Bigrams (pairs of words)
-
🧬 MeSH Terms (Medical Subject Headings)
These features are extracted after stopwords are removed and remaining terms are stemmed.
Although we published a paper on a different classifier (Springer article), we opted not to use that model in Rayyan. Some of the features in that study, such as co-citations, are difficult to incorporate in a real-time production system.
📌 Note: We're currently working on a new publication focused entirely on Rayyan and its unique approach.
⚙️ How It Learns
As you (and your team) make inclusion/exclusion decisions, Rayyan's classifier starts to learn from your patterns. Once you’ve made at least:
-
✅ 50 screening decisions
-
With minimum 5 "Include" and 5 "Exclude"
-
…the system trains the model to classify the remaining undecided articles.
It then calculates a confidence score for each unscreened article — based on how similar it is to your previous decisions — and translates it into a thumbs up (Include) or thumbs down (Exclude) rating.
🔁 Continuous Improvement
As you keep screening, Rayyan automatically re-evaluates its model. If it detects that new labeled examples could improve its predictions, it retrains the classifier and re-scores the remaining undecided citations. This process continues until:
- 🗃️ All citations are labeled
- 🧠 The model reaches optimal performance and can't be further improved
📊 Validated Performance
Rayyan’s classifier was tested using the same features in a previous study:
➡️ Read the full JAMIA study
Study Highlights:
-
📚 15 systematic review datasets
-
🧪 2-fold cross-validation (repeated 10 times, with 50% training / 50% testing)
-
🧮 Metrics used:
-
AUC (Area Under the Curve): 0.87 ± 0.09
-
WSS@95 (Work Saved over Random Sampling): 0.49 ± 0.18
-
WSS@95 tells you how many citations reviewers didn’t have to screen thanks to the classifier, while still maintaining 95% recall.
📘 Want to Dive Deeper?
You can check out more technical details in our original publication:
➡️ Machine Learning for Systematic Reviews - Springer Article
While that study used Random Forest and extra features (like co-citations), those methods aren’t feasible for real-time use within Rayyan — which is why we rely on our current, streamlined SVM-based model.
💬 Still Need Help?
We’re here for you. Submit a support ticket and we’ll assist you personally.
And don’t forget to follow us on Twitter @rayyanapp for updates, tips, and tricks!
Comments
0 comments
Article is closed for comments.