AI for Research Proposal Automation
A research-to-proposal workflow—and a close look at what happens when fluent drafts and confident evaluations outrun their evidence.
Scope: final project for Designing Advanced Generative AI Workflows, submitted as part of my Johns Hopkins Applied Generative AI studies. This is an executed academic notebook using supplied research documents and a funding call, not a grant submission, clinical trial, or deployed product.
The problem
How can a researcher connect a large publication archive to a specific funding opportunity without mistaking a related topic for evidence that actually supports a proposal? The assignment used a mental-health funding call to explore retrieval, research screening, ideation, drafting, and review.
Executed workflow
- Extract funding topic
- Retrieve candidate papers
- Screen evidence
- Generate and select ideas
- Draft proposal
- Evaluate and refine
What ran
- 112 PDFs → 9,111 chunks: PDF loading and 800-character chunks with 150-character overlap.
- 120 retrieved chunks → 12 paper candidates: all-MiniLM-L6-v2 embeddings and a FAISS index narrowed the corpus for detailed model screening.
- 4 sources retained → 5 ideas → 1 draft: gpt-4o-mini extracted the funding topic, screened the candidates, generated ideas, selected one, and drafted against the supplied template.
- Structured evaluation and written human review: the same model family scored the original proposal; I then reviewed both the proposal and the critique and wrote a conditional revision.
These are observed run counts, not retrieval-accuracy or productivity metrics. Only 12 of 112 papers received detailed screening.
Follow one evidence trail
- Funding topic extracted
- “Digital mental health intervention optimization and effectiveness testing.”
- Source and screening
- The supplied observational social-media/mental-health paper was retained as BACKGROUND, with PDF pages 1–5 recorded as its evidence locator. That is not evidence of treatment efficacy.
- Generated idea
- “Optimizing Video Content in Digital Mental Health Interventions.” The draft proposed video-versus-text delivery and six-week anxiety and depression measures.
- Where it failed
- The draft introduced two references unsupported by the supplied evidence and treated an observational motivation as stronger support than it was. A plausible proposal structure did not establish a valid intervention.
- My revision
- Removed the unsupported references; made the platform partner, evidence base, and 1,000-active-user requirement unresolved dependencies; specified a proposed primary six-week depression endpoint and active treatment comparator; omitted heart-rate variability without a validation argument.
Those are revisions to a research concept, not completed clinical work. No partnership, approval, recruitment, efficacy, or adequate statistical power is claimed.
The judge needed review, too
The original draft received model scores of 3/5 Innovation, 3/5 Significance, 2/5 Approach, and 3/5 Investigator Expertise—a 2.75/5 mean across four criteria for one draft. The assignment defined 1 as poor and 5 as excellent. The gpt-4o-mini judge used the draft, selected source, and funding call at temperature 0.
One critique said the draft claimed a completed power analysis. It did not. My review identified wording in the evaluation prompt that named that anticipated fault; the model repeated it as a finding while missing the unsupported references. This run suggests a prompt-induced failure, not a controlled causal test.
The accurate criticism was missing sample-size justification and planning assumptions. The useful lesson: check the evaluator against the source, not just the draft against the evaluator.
These scores belong only to the original model-generated draft—not the revised concept, my course grade, or an independently validated quality assessment. The revision was not rerun or rescored.
Limits worth keeping visible
- Duplicate versions of one paper received inconsistent include/exclude decisions. The model also labeled social-media causal analysis as DIRECT evidence despite the stricter treatment-evidence definition.
- PDF extraction warnings, context truncation, and limited retrieval coverage leave room for missed or distorted evidence.
- Neither a well-formed category nor valid evaluation JSON guarantees factual accuracy. Bibliographic details and source support still need independent checks.
- No retrieval benchmark, time-savings study, grant outcome, or clinical benefit was measured.
What I would improve next
Deduplicate before screening; verify citations against exact passages; justify inclusion against the funding criteria; test alternative retrieval queries; and separately check whether the judge’s criticisms describe anything the draft actually says. Clinical and statistical experts would need to review any real research protocol.
Source: my submitted “JHU AI for Research Proposal Automation Final Project” HTML notebook, including the preserved executed outputs and final written review. This public case study summarizes that evidence; raw notebooks, credentials, and source files are not published.