4  Guide 1 — AI Research Discovery Agent

4.1 “Your AI Research Scientist”

4.2 The problem

Researchers spend months finding:

  • novel research gaps,
  • the right papers among thousands,
  • emerging topics and methods,
  • the key authors and groups in a field.

Most of them still read hundreds of papers manually, keep notes in Word files, and hope their idea is novel.

4.3 The solution

An autonomous agent that, given a one-line research interest, continuously monitors the literature (arXiv, PubMed, Semantic Scholar, Crossref — extendable to IEEE Xplore, Springer, Elsevier, Google Scholar) and builds a personal research-intelligence system:

  1. Literature Mining Agent — finds the latest N papers, builds a citation network, finds key authors and trends.
  2. Research Gap Agent — reads the abstracts and writes explicit gap statements (“Existing studies predict drought stress using RGB imaging, but few integrate hyperspectral + genomic data”).
  3. Idea Generator Agent — proposes ranked PhD/grant topics with novelty and feasibility scores.
  4. Proposal Writer Agent — turns the best idea into a research proposal: title, novelty statement, research questions, hypotheses, objectives, methodology sketch, timeline.

4.4 Example

Input

“I want to work on AI-based drought prediction in wheat.”

Output (Markdown report + JSON)

  • 50–500 papers with IDs, years, venues, citation counts
  • Trend summary (topics per year, rising methods)
  • 5–8 research gaps with supporting paper IDs
  • 5 ranked topic ideas, e.g. “Multimodal AI for drought-resilience prediction”, “Explainable ML for genomic selection”
  • A 2–3 page research proposal for the top idea

4.5 Architecture

topic ──► [Search tools] arXiv / Semantic Scholar / PubMed / Crossref
              │  (real APIs, free)
              ▼
        Literature Mining Agent   → PaperSet (Pydantic), trends, key authors
              │
              ▼
        Research Gap Agent        → GapAnalysis (list of gaps + evidence IDs)
              │
              ▼
        Idea Generator Agent      → IdeaList (title, novelty, feasibility, impact)
              │
              ▼
        Proposal Writer Agent     → proposal.md

Key design rules:

  • Citations only from tool results. The model receives the retrieved papers and is instructed to cite only their IDs.
  • Structured outputs (Pydantic) for papers, gaps and ideas so they can be rendered as tables or stored in a DB.
  • Idempotent runs: every run writes outputs/<slug>/papers.json, gaps.json, ideas.json, proposal.md.

4.6 Build steps (what the code does)

Step File What you learn
1 tools.py Calling arXiv, Semantic Scholar, PubMed & Crossref; normalising results into one Paper model
2 llm.py A thin Claude client with streaming + structured outputs
3 agents.py Four single-responsibility agents with clear system prompts
4 main.py A CLI orchestrator (--topic, --max-papers, --years) that runs the pipeline and saves outputs

Run:

python agent_01_research_discovery\main.py --topic "AI-based drought prediction in wheat" --max-papers 60

4.7 Extending it (product features)

  • Continuous monitoring: schedule weekly runs (cron / Managed Agents scheduled deployments) and diff the new papers → “What’s new this week in your field” email.
  • Citation network graph: use Semantic Scholar’s references/citations endpoints + networkx → find the “bridge” papers.
  • Personal library: store papers in SQLite/Postgres + vector index for RAG-style Q&A.
  • Web UI: Streamlit for the MVP, then FastAPI + React.
  • More sources: Scopus/IEEE/Springer via institutional API keys (sold as an institutional feature).

4.8 Market value

Customer Pain Willing to pay
PhD students “Is my topic novel?” $10–30/month
Supervisors / PIs Need topics for new students, grant ideas $30–50/month or lab plan
Universities / ORICs Improve proposal quality & research output $20k–100k/year
R&D departments Technology scouting, competitor tracking $50k–200k/year

Estimated TAM for discovery tools alone: $1–2B (research-intelligence + scholarly search market).

4.9 How to sell it

  1. Free “Is my topic novel?” check on the website — users paste a topic, get 3 gaps and 3 ideas (watermarked). Capture email.
  2. Content marketing — YouTube walkthroughs (“I found a PhD topic in 10 minutes”), WhatsApp community, university seminars.
  3. Supervisor referrals — supervisors get a free seat for every 5 students.
  4. ORIC pilots — offer a department-wide 3-month pilot; report “proposals generated / accepted”.
  5. Upsell to the Literature Review Agent (natural next step) and the full Research OS.

4.10 Risks & mitigations

Risk Mitigation
Hallucinated papers Only tool-retrieved papers may be cited; verify DOIs via Crossref
Shallow gap analysis Feed abstracts (not just titles); allow full-text PDFs in Pro tier
API rate limits Cache results; polite delays; institutional API keys
“Ideas are generic” Ask for feasibility, required data, required equipment per idea; score novelty against retrieved set

4.11 90-day roadmap

  • Weeks 1–2: CLI (this repo) → Weeks 3–6: Streamlit UI + PDF upload → Weeks 7–10: weekly monitoring & email digests → Weeks 11–13: citation graph & lab plan.

Codanics — Python ka Chilla: Build AI Agents

  • Course: https://codanics.com/courses/python-ka-chilla-build-ai-agents/
  • Website: https://codanics.com · YouTube: @codanics
  • WhatsApp channel: https://whatsapp.com/channel/0029Va7nRDq3QxRzGqaQvS3r
  • Complete free course: https://www.youtube.com/watch?v=UE3o-7KRNXs&list=PLaQmdklCcX40