Once your jury round is open and jurors are scoring, you need to track progress, identify lagging jurors, and ensure the round completes on time. This guide covers everything you need to monitor a live jury session.
The Jury Management Dashboard
Navigate to Calls → [Your Call] → Jury. The dashboard refreshes automatically every 15 seconds, giving you a near-real-time view of scoring activity.
Summary Cards
At the top, three summary cards show aggregate data:
Triage Summary: If using triage voting, shows the top-ranked submissions by yes/maybe/no votes. Click Export CSV to download the full triage data.
Ranking Summary: If using ranking mode, shows mean rank positions.
Rubric Summary: If using rubric scoring, shows score averages per criterion. Toggle Raw Scores vs normalized scores to account for juror bias.
Stats at a Glance
Four stat cards show:
Total Submissions under review
Jury Members assigned and active
Categories (if using category-based scoring)
Jury Rounds created
Tracking Per-Juror Completion
Viewing Assignment Progress
On the Jury Members tab, each juror card shows:
Their name and email
When they were assigned
Their scoring progress (scored / total assigned)
In DISTRIBUTED or MANUAL assignment modes, each juror has a different set of submissions. In ALL mode, every juror reviews every submission.
Identifying Lagging Jurors
Look for jurors with low completion percentages as the deadline approaches. Common reasons jurors fall behind:
They haven't accepted their invitation yet (check invitation status)
They started but got stuck on a submission
They're waiting until the last minute
Sending Reminders
Crafted Call automatically sends scoring reminders at configurable intervals (typically 7 days, 3 days, 1 day, and 4 hours before the round deadline). These are deduplicated — each juror receives each reminder level only once.
To manually follow up with a specific juror:
Click the mail icon next to their name on the Jury Members tab
Was this article helpful?
This opens your email client with their address pre-filled
Reference the call name and deadline in your message
Understanding Scoring Results in Real Time
The Scores Tab
As jurors submit scores, you can see results building in real time:
Per-submission averages: The mean score across all jurors who have scored that submission
Score count: How many jurors have scored each submission (critical for identifying under-reviewed submissions)
Score distribution: High vs low scores to spot consensus or disagreement
Score Normalization
If some jurors consistently score higher or lower than others, enable score normalization on the Rubric Summary card. This adjusts for juror bias using z-score normalization, ensuring a "tough grader" and a "generous grader" contribute equally to the final ranking.
When to use normalization:
3 or more jurors
You notice significant scoring range differences between jurors
You want the fairest possible ranking
Handling Common Issues During a Live Round
A Juror Hasn't Started Scoring
Check if they accepted their invitation (Jury Members tab shows status)
If the invitation is still pending, click Resend Invitation
Contact them directly via email
A Juror Needs to Drop Out
If a juror can no longer participate:
Go to the Jury Members tab
Click the remove button next to their name
Confirm the removal
What happens:
All their existing scores for this round are deleted
In DISTRIBUTED mode, their unscored submissions are automatically redistributed to remaining jurors
In ALL mode, submissions simply have fewer reviews (the remaining jurors' scores still count)
An audit log entry records the removal
Important: Once a juror is removed, their scores cannot be recovered. If you only want to pause their access, consider leaving them assigned and communicating directly.
Scoring Seems Inconsistent
If one juror's scores look unusual:
Check the Rubric Summary with normalization enabled — this shows if they're systematically higher or lower
Use the per-juror breakdown to compare their scoring pattern
If a specific score is clearly erroneous, you can invalidate individual scores from the Scores tab (the score is preserved for audit but excluded from calculations)
Invalidated scores can be reinstated if needed
The Deadline is Approaching and Scoring is Incomplete
Options:
Extend the round: Edit the round end date to give jurors more time
Close anyway: If enough scores exist for a fair decision, you can transition the round to SCORING_COMPLETE even with some jurors unfinished
The system will warn you about incomplete scoring when you try to close
Audit Trail
Every significant action is logged in the Audit Trail tab:
Juror assignments and removals
Score submissions and edits
Round status changes
Score invalidations and reinstatements
Bulk decision actions
Use this for post-show documentation, compliance records, or resolving disputes about the jury process. The audit trail cannot be edited or deleted.
Timeline Planning Tips
Show Size
Jurors
Recommended Round Length
Under 100 submissions
3 jurors
5-7 days
100-300 submissions
3-5 jurors
7-14 days
300-500 submissions
5-7 jurors
14-21 days
500+ submissions
7+ jurors
21-30 days
Rule of thumb: A juror can review approximately 30-50 submissions per hour using rubric scoring (fewer if providing detailed feedback). Plan accordingly and communicate expected time commitment when inviting jurors.
