Understanding Blinding, Labels, Reasons, and Ratings in Collaborative Reviews

When working with others on a Rayyan review, it’s important to understand how blinding, labels & reasons, and ratings behave. Here’s everything you need to know!

🔒 What is Blinding in Collaborative Reviews?

By default, all reviews in Rayyan start with blinding ON to reduce bias among team members. Here's how it works:

• Blinding ON (default):

  • Decisions (include/exclude/maybe), labels, exclusion reasons, and notes added by one member are hidden from all other collaborators.

  • This setting helps keep each member’s assessments independent.

• Blinding OFF:

  • All team members can see each other's decisions, labels, reasons, and notes in real-time.

  • Ideal when you want full visibility into each other’s work for coordination or final review stages.

Note: Only the review owner can toggle blinding. In Team Reviews, the Team Owner and Admins can control blinding.

🏷️ How Do Labels and Exclusion Reasons Work in Teams?

Each team member can add their own labels or reasons during screening — and here’s how they’re managed:

• With blinding ON, each collaborator’s labels and reasons are hidden from others.

• If your team wants to use a standardized list, it’s best to share it via review chat or your preferred communication channel.

• Once a label or reason is added to any reference, it becomes available in the dropdown list for future use by that user.

• All applied labels and reasons will appear under their respective facets in the Filters workbench to help you track and filter references.

⚠️ Typos or formatting differences (like Wrong age vs wrong age) will create separate entries and cannot be merged.

For editing or deleting a label/reason, check out this guide:
🔗 How Can I Edit or Delete a Label or Exclusion Reason in Rayyan?

📊 How Are Ratings Computed in Collaborative Reviews?

Rayyan’s AI Ratings feature helps estimate how well your decisions align with your past screening — and yes, it still works in collaborative reviews, even when blinding is ON.

Here’s how it works:

• Ratings are calculated based on all screening decisions made by the team — whether you’re working blinded or not.

• Let’s say:

  • Member A screens 30 references

  • Member B screens 20

  • Member C screens 30

  • Member D screens 50
    Out of a total of 600 references, the AI will analyze all these decisions together.

• If multiple members screen the same reference and make conflicting decisions (e.g., one includes, the other excludes), Rayyan will disregard this reference and won’t use such entries when computing ratings to avoid confusion.

📌 The more consistent and comprehensive your screening is, the better the AI’s predictions will be for unscreened references.

For a deeper dive into how ratings work, visit this guide:
🔗 How to use Rayyan's Predictions Classifier for Relevance Ranking?