AI Inbox Triage & Drafting Assistant
A human-reviewed system for turning a crowded inbox into prioritized work, grounded summaries, and safer first-response drafts.
Scope: midterm project for Applied Generative AI study at Johns Hopkins, submitted under the assignment title AI Email Secretary. It uses a synthetic business scenario and sample email dataset; it is not connected to a live inbox and does not send messages autonomously.
Workflow
- Sample inbox
- Previous-day filter
- Classify priority
- Summarize and draft
- Evaluate against source
- Human review
Problem
A manager facing a mixed inbox needs to separate genuine emergencies and same-day deadlines from routine updates, informational messages, and spam. A useful assistant must reduce review time without mistaking urgent language for actual urgency—or making commitments on the manager’s behalf.
Approach
- Filter by fixed rules. Apply the Yesterbox method with pandas so only the previous day’s messages enter the workflow.
- Classify with guardrails. Sort messages into six defined categories while checking the full body, recipient, context, and actual deadline.
- Generate one item at a time. Produce source-grounded summaries, next steps, and first-response drafts for each critical email.
- Protect consequential decisions. Prevent the model from inventing approvals, commitments, attendance, attachment reviews, or unsupported facts.
- Evaluate against the source. Use a separate evaluation call to score each draft for relevance, clarity, and actionability, with every score traceable to the original email.
Result
The workflow processed all 51 previous-day messages in the sample dataset. It identified 23 critical messages, generated complete summaries for 13 urgent and 10 deadline-driven emails, produced 23 first-response drafts, and evaluated all 23 drafts against their source messages.
Model-evaluated draft scores, n=23. The generator and judge both used gpt-4o-mini; these are not independent accuracy measurements.
- Relevance: 4.00 / 5 average
- Clarity: 4.70 / 5 average
- Actionability: 3.09 / 5 average
The lower actionability score exposed the project’s most useful next step: make safe follow-up timing, ownership, and requests for missing information more specific without allowing the model to fabricate a decision.
One example from the run
Actual synthetic coursework example, email ID 1. Sender details omitted; input summarized, draft excerpt quoted.
- Input
- A budget-approval request with an end-of-day deadline.
- Classification
- Deadline-Driven Emails.
- Draft excerpt
- “I will review the details and confirm my decision as soon as possible.”
- Evaluator note
- “While the response indicates a review will take place, it lacks a specific timeline for when the decision will be made.” Actionability: 3/5.
The draft does not grant the approval. But “as soon as possible” is vague: a person still needs to set a realistic follow-up and verify the deadline. The source email was received March 3 and said “by the end of today,” while the workflow’s review date was March 4. That unresolved date mismatch is another reason not to accept the classification as verified ground truth.
Evaluation method
Each original email and its draft went into a separate gpt-4o-mini judge call at temperature 0. The 1–5 rubric assessed relevance (accuracy and coverage), clarity (understandable, professional communication), and actionability (a safe, concrete next step). Five is the top of the scale, not proof of correctness; the prompt did not supply calibrated score-by-score anchors. All 23 drafts were scored. Human review remains required before use; no independent benchmark or measured time savings is claimed.
Limits
The assistant supports triage and drafting; it does not make or send decisions. A person must verify classifications, facts, dates, and commitments before any response is sent.