16  Code: Agent 1 — Research Discovery

Folder: 02_agents_code/agent_01_research_discovery/ — self-contained; run python agent_01_research_discovery/main.py --help.

16.1 README

16.1.1 README.md

# Agent 1 — AI Research Discovery Agent ("Your AI Research Scientist")

Turns a one-line research interest into: retrieved papers → trend report → research gaps → ranked ideas → a research proposal.

```
topic ─► LiteratureMiningAgent (plan queries, analyse trends)
      ─► search tools: arXiv · Semantic Scholar · PubMed · Crossref   (real, free APIs)
      ─► ResearchGapAgent  (structured gaps with evidence IDs)
      ─► IdeaGeneratorAgent (5 ranked ideas: novelty / feasibility / impact)
      ─► ProposalWriterAgent (proposal.md)
```

## Files

| File | Purpose |
|------|---------|
| `main.py` | CLI orchestrator (`--topic`, `--max-papers`, `--sources`, `--offline`) |
| `agents.py` | The four sub-agents + their Pydantic output models |
| `tools.py` | Search functions for 4 databases, a common `Paper` model, de-duplication, offline sample |
| `llm.py` | Thin Claude client: `ask()` (streamed text) and `ask_structured()` (JSON → Pydantic) |
| `outputs/` | One folder per topic: `papers.json`, `trends.json`, `gaps.json`, `ideas.json`, `proposal.md` |

## Run

```powershell
# from 02_agents_code with the .venv activated and ANTHROPIC_API_KEY in .env
python agent_01_research_discovery\main.py --topic "AI-based drought prediction in wheat"
python agent_01_research_discovery\main.py --topic "..." --max-papers 60 --sources arxiv,semantic_scholar
python agent_01_research_discovery\main.py --topic "..." --offline      # no internet needed for search
```

## What this agent teaches

1. **Tool use with real APIs** — normalising four different APIs into one `Paper` model.
2. **Structured outputs** — `ask_structured()` forces JSON that matches a Pydantic model.
3. **Citation grounding** — the model may only cite `[ids]` present in its input.
4. **Pipeline orchestration** — small agents, each with one job, chained in `main.py`.

## Ideas to extend

- Weekly scheduled runs + "what's new" diff → email digest.
- Citation network (Semantic Scholar references/citations) + `networkx` centrality.
- Streamlit UI; save papers to SQLite; RAG-style Q&A over abstracts.

---

**Codanics — Python ka Chilla: Build AI Agents**
Course: https://codanics.com/courses/python-ka-chilla-build-ai-agents/ · Website: https://codanics.com · YouTube: [@codanics](https://www.youtube.com/@codanics)
WhatsApp: https://whatsapp.com/channel/0029Va7nRDq3QxRzGqaQvS3r · Free course: https://www.youtube.com/watch?v=UE3o-7KRNXs&list=PLaQmdklCcX40

16.2 Source files

16.2.1 agents.py

"""
agents.py — the four sub-agents of the Research Discovery Agent.

    1. LiteratureMiningAgent  : query expansion + trend / key-author analysis
    2. ResearchGapAgent       : explicit, evidence-linked gap statements
    3. IdeaGeneratorAgent     : ranked research / PhD topic ideas
    4. ProposalWriterAgent    : a research proposal for the best idea

Each agent = one system prompt + one (structured or free-text) model call.
This "small, single-responsibility agents" style is easy to test and easy to teach.

Codanics — Python ka Chilla: Build AI Agents
https://codanics.com/courses/python-ka-chilla-build-ai-agents/
"""
from __future__ import annotations

from pydantic import BaseModel, Field

from llm import ask, ask_structured
from tools import Paper

BRAND = ("Codanics — Python ka Chilla: Build AI Agents | "
         "https://codanics.com/courses/python-ka-chilla-build-ai-agents/")

CITE_RULE = (
    "CITATION RULE: You may cite ONLY the papers listed in the input, using their exact [id] tags "
    "(e.g. [arxiv:2401.01234], [pmid:38012345], [doi:10.1000/xyz]). Never invent papers, authors or IDs."
)


# ============================================================ 1. Literature Mining
class SearchPlan(BaseModel):
    queries: list[str] = Field(description="3-6 database search queries (boolean/keyword style)")
    key_concepts: list[str]


class TrendReport(BaseModel):
    summary: str = Field(description="4-6 sentence overview of the field based on the papers")
    themes: list[str] = Field(description="Main research themes with cited [ids]")
    rising_methods: list[str] = Field(description="Methods/technologies gaining traction, with [ids]")
    key_authors_or_groups: list[str]
    data_sources_used: list[str] = Field(description="Datasets / data types used in the literature")


class LiteratureMiningAgent:
    SYSTEM = f"You are a senior research librarian and scientometrics expert. {CITE_RULE}"

    def plan(self, topic: str) -> SearchPlan:
        return ask_structured(
            self.SYSTEM,
            f"Research interest: {topic}\n\nDesign search queries that maximise recall across arXiv, "
            f"PubMed, Semantic Scholar and Crossref. Include synonyms and adjacent methods.",
            SearchPlan, effort="low")

    def analyse(self, topic: str, papers: list[Paper]) -> TrendReport:
        corpus = "\n".join(p.short() for p in papers)
        return ask_structured(
            self.SYSTEM,
            f"Topic: {topic}\n\nPapers:\n{corpus}\n\nAnalyse themes, rising methods, key authors and data sources.",
            TrendReport)


# ============================================================== 2. Research Gap
class Gap(BaseModel):
    statement: str = Field(description="One precise gap statement, e.g. 'Existing studies predict drought "
                                       "stress using RGB imaging, but few integrate hyperspectral + genomic data.'")
    evidence_ids: list[str] = Field(description="Paper [ids] that support this gap")
    why_it_matters: str
    difficulty: str = Field(description="low | medium | high")


class GapAnalysis(BaseModel):
    gaps: list[Gap]


class ResearchGapAgent:
    SYSTEM = (f"You are a critical research strategist who finds what has NOT been done. {CITE_RULE} "
              "Gaps must be specific (data, method, population, scale, validation), not generic.")

    def run(self, topic: str, papers: list[Paper], trends: TrendReport) -> GapAnalysis:
        corpus = "\n".join(p.short() for p in papers)
        return ask_structured(
            self.SYSTEM,
            f"Topic: {topic}\n\nTrend report:\n{trends.model_dump_json(indent=1)}\n\nPapers:\n{corpus}\n\n"
            f"Identify 5-8 concrete research gaps.",
            GapAnalysis, effort="high")


# ============================================================ 3. Idea Generation
class Idea(BaseModel):
    title: str
    one_paragraph: str
    addresses_gaps: list[int] = Field(description="Indices (0-based) of the gaps this idea addresses")
    novelty_score: int = Field(ge=1, le=10)
    feasibility_score: int = Field(ge=1, le=10)
    impact_score: int = Field(ge=1, le=10)
    required_data: list[str]
    required_skills: list[str]
    suitable_for: str = Field(description="MS thesis | PhD | grant | industry R&D")


class IdeaList(BaseModel):
    ideas: list[Idea]


class IdeaGeneratorAgent:
    SYSTEM = ("You are an experienced PhD supervisor and grant reviewer. Propose research topics that are "
              "novel, feasible in 2-4 years and clearly linked to the identified gaps. " + CITE_RULE)

    def run(self, topic: str, gaps: GapAnalysis) -> IdeaList:
        return ask_structured(
            self.SYSTEM,
            f"Topic: {topic}\n\nGaps:\n{gaps.model_dump_json(indent=1)}\n\nPropose 5 ranked ideas (best first).",
            IdeaList, effort="high")


# ============================================================ 4. Proposal Writer
class ProposalWriterAgent:
    SYSTEM = ("You write concise, rigorous research proposals in Markdown for university review committees. "
              + CITE_RULE + " Use headings: Title, Background, Novelty Statement, Research Questions, "
              "Hypotheses, Objectives, Methodology, Expected Outcomes, Timeline (Gantt-style table), "
              "Risks & Mitigations, References (only cited ids).")

    def run(self, topic: str, idea: Idea, gaps: GapAnalysis, papers: list[Paper], echo: bool = False) -> str:
        corpus = "\n".join(p.short(400) for p in papers)
        body = ask(
            self.SYSTEM,
            f"Research interest: {topic}\n\nSelected idea:\n{idea.model_dump_json(indent=1)}\n\n"
            f"Gaps:\n{gaps.model_dump_json(indent=1)}\n\nPapers available for citation:\n{corpus}\n\n"
            f"Write a 2-3 page proposal.",
            effort="high", echo=echo)
        return body + f"\n\n---\n*Generated with the AI Research Discovery Agent — {BRAND}*\n"

16.2.2 llm.py

"""
llm.py — a thin wrapper around the Anthropic Claude API used by this agent.

Two functions are all you need:
    ask(system, user)                  -> free text (streamed)
    ask_structured(system, user, Model) -> a validated Pydantic object (JSON structured output)

Why a wrapper?  Every agent in this project talks to the model the same way.  Keeping the
API details in ONE file per agent means you can swap the model or provider in one place.

Codanics — Python ka Chilla: Build AI Agents
https://codanics.com/courses/python-ka-chilla-build-ai-agents/
"""
from __future__ import annotations

import os
from typing import Type, TypeVar

import anthropic
from dotenv import load_dotenv
from pydantic import BaseModel

load_dotenv()  # reads ANTHROPIC_API_KEY / CLAUDE_MODEL from a .env file if present

MODEL = os.getenv("CLAUDE_MODEL", "claude-opus-5")
T = TypeVar("T", bound=BaseModel)

_client: anthropic.Anthropic | None = None


def client() -> anthropic.Anthropic:
    """Lazily create one shared client (credentials come from the environment)."""
    global _client
    if _client is None:
        _client = anthropic.Anthropic()
    return _client


def _check(msg: anthropic.types.Message) -> None:
    if msg.stop_reason == "refusal":
        detail = getattr(msg, "stop_details", None)
        raise RuntimeError(f"The model refused this request: {detail}")
    if msg.stop_reason == "max_tokens":
        raise RuntimeError("Output was cut off (max_tokens). Increase max_tokens and retry.")


def _text(msg: anthropic.types.Message) -> str:
    return "".join(b.text for b in msg.content if b.type == "text")


def ask(system: str, user: str, *, max_tokens: int = 16000, effort: str = "medium",
        echo: bool = False) -> str:
    """Free-text generation. Streams so long outputs never hit HTTP timeouts."""
    with client().messages.stream(
        model=MODEL,
        max_tokens=max_tokens,
        system=system,
        messages=[{"role": "user", "content": user}],
        thinking={"type": "adaptive"},
        output_config={"effort": effort},
    ) as stream:
        if echo:
            for chunk in stream.text_stream:
                print(chunk, end="", flush=True)
            print()
        msg = stream.get_final_message()
    _check(msg)
    return _text(msg)


def ask_structured(system: str, user: str, model_cls: Type[T], *, max_tokens: int = 16000,
                   effort: str = "medium") -> T:
    """Structured generation: the SDK converts the Pydantic model to a JSON schema, the API
    constrains the output to it, and `parsed_output` is a validated instance of `model_cls`."""
    with client().messages.stream(
        model=MODEL,
        max_tokens=max_tokens,
        system=system,
        messages=[{"role": "user", "content": user}],
        thinking={"type": "adaptive"},
        output_config={"effort": effort},
        output_format=model_cls,
    ) as stream:
        msg = stream.get_final_message()
    _check(msg)
    if msg.parsed_output is None:  # should not happen with a constrained schema; be defensive
        return model_cls.model_validate_json(_text(msg))
    return msg.parsed_output

16.2.3 main.py

"""
AI Research Discovery Agent — "Your AI Research Scientist"
===========================================================
topic ─► search (arXiv, Semantic Scholar, PubMed, Crossref) ─► trends ─► gaps ─► ideas ─► proposal

Usage
    python main.py --topic "AI-based drought prediction in wheat"
    python main.py --topic "..." --max-papers 60 --sources arxiv,semantic_scholar
    python main.py --topic "..." --offline          # demo without internet (small built-in sample)

Outputs are written to outputs/<topic-slug>/  (papers.json, trends.json, gaps.json, ideas.json, proposal.md)

Codanics — Python ka Chilla: Build AI Agents
https://codanics.com/courses/python-ka-chilla-build-ai-agents/
"""
from __future__ import annotations

import json
import sys
from pathlib import Path

import typer
from rich.console import Console
from rich.panel import Panel
from rich.table import Table
from slugify import slugify

sys.path.insert(0, str(Path(__file__).parent))
if hasattr(sys.stdout, "reconfigure"):  # Windows consoles default to cp1252; we print → ✓ etc.
    sys.stdout.reconfigure(encoding="utf-8")  # allow `python path/to/main.py`
from agents import (IdeaGeneratorAgent, LiteratureMiningAgent, ProposalWriterAgent,  # noqa: E402
                    ResearchGapAgent)
from tools import OFFLINE_SAMPLE, Paper, dedupe, search_all  # noqa: E402

app = typer.Typer(add_completion=False)
console = Console()
OUT_DIR = Path(__file__).parent / "outputs"

BANNER = """[bold cyan]AI Research Discovery Agent[/] — Your AI Research Scientist
[dim]Codanics · Python ka Chilla: Build AI Agents · https://codanics.com/courses/python-ka-chilla-build-ai-agents/
YouTube @codanics · WhatsApp https://whatsapp.com/channel/0029Va7nRDq3QxRzGqaQvS3r[/]"""


def save(path: Path, data) -> None:
    path.parent.mkdir(parents=True, exist_ok=True)
    if isinstance(data, str):
        path.write_text(data, encoding="utf-8")
    else:
        path.write_text(json.dumps(data, indent=2, ensure_ascii=False), encoding="utf-8")
    console.print(f"  [green]saved[/] {path.relative_to(Path(__file__).parent)}")


@app.command()
def run(
    topic: str = typer.Option(..., help="Your research interest in one sentence"),
    max_papers: int = typer.Option(40, help="Approximate number of papers to analyse"),
    sources: str = typer.Option("arxiv,semantic_scholar,pubmed,crossref", help="Comma-separated sources"),
    offline: bool = typer.Option(False, help="Use the built-in demo papers instead of live search"),
    echo: bool = typer.Option(True, help="Stream the proposal to the console while it is written"),
):
    console.print(Panel(BANNER, expand=False))
    out = OUT_DIR / slugify(topic)[:60]
    miner, gapper, ideator, writer = (LiteratureMiningAgent(), ResearchGapAgent(),
                                      IdeaGeneratorAgent(), ProposalWriterAgent())

    # 1) Search ------------------------------------------------------------------
    if offline:
        console.rule("[bold]1. Literature mining (offline demo)")
        papers: list[Paper] = OFFLINE_SAMPLE
    else:
        console.rule("[bold]1. Literature mining")
        plan = miner.plan(topic)
        console.print(f"  queries: {plan.queries}")
        src = [s.strip() for s in sources.split(",") if s.strip()]
        per_source = max(5, max_papers // (len(plan.queries) * len(src)) + 2)
        found: list[Paper] = []
        for q in plan.queries:
            console.print(f"  [cyan]query[/]: {q}")
            found.extend(search_all(q, per_source, src, log=lambda m: console.print(f"    {m}")))
        papers = dedupe(found)[:max_papers]
        if not papers:
            console.print("[yellow]No papers retrieved (network?). Falling back to offline demo sample.[/]")
            papers = OFFLINE_SAMPLE
    save(out / "papers.json", [p.model_dump() for p in papers])

    t = Table(title=f"{len(papers)} papers", show_lines=False)
    t.add_column("id"); t.add_column("year"); t.add_column("title", max_width=80)
    for p in papers[:15]:
        t.add_row(p.id, str(p.year or ""), p.title)
    console.print(t)

    # 2) Trends ------------------------------------------------------------------
    console.rule("[bold]2. Trend analysis")
    trends = miner.analyse(topic, papers)
    console.print(trends.summary)
    save(out / "trends.json", trends.model_dump())

    # 3) Gaps --------------------------------------------------------------------
    console.rule("[bold]3. Research gaps")
    gaps = gapper.run(topic, papers, trends)
    for i, g in enumerate(gaps.gaps):
        console.print(f"  [bold]{i}.[/] {g.statement}  [dim]{g.evidence_ids}[/]")
    save(out / "gaps.json", gaps.model_dump())

    # 4) Ideas -------------------------------------------------------------------
    console.rule("[bold]4. Research ideas")
    ideas = ideator.run(topic, gaps)
    t = Table(title="Ranked ideas")
    for col in ("#", "title", "novelty", "feasibility", "impact", "suitable for"):
        t.add_column(col)
    for i, idea in enumerate(ideas.ideas, 1):
        t.add_row(str(i), idea.title, str(idea.novelty_score), str(idea.feasibility_score),
                  str(idea.impact_score), idea.suitable_for)
    console.print(t)
    save(out / "ideas.json", ideas.model_dump())

    # 5) Proposal ----------------------------------------------------------------
    console.rule("[bold]5. Research proposal (best idea)")
    proposal = writer.run(topic, ideas.ideas[0], gaps, papers, echo=echo)
    save(out / "proposal.md", proposal)
    console.print(Panel(f"Done. All outputs in [bold]{out}[/]", style="green"))


if __name__ == "__main__":
    app()

16.2.4 tools.py

"""
tools.py — real literature-search tools for the Research Discovery Agent.

Every function returns a list[Paper] in ONE common shape, no matter which database it came from.
All four sources are free, public APIs (no key needed; keys only raise rate limits):

    arXiv            https://arxiv.org/help/api
    Semantic Scholar https://api.semanticscholar.org/
    PubMed           https://www.ncbi.nlm.nih.gov/books/NBK25501/  (E-utilities)
    Crossref         https://api.crossref.org/

Golden rule of this project: the LLM may only cite papers that came out of these tools.

Codanics — Python ka Chilla: Build AI Agents
https://codanics.com/courses/python-ka-chilla-build-ai-agents/
"""
from __future__ import annotations

import os
import re
import time
import xml.etree.ElementTree as ET
from typing import Iterable

import requests
from dotenv import load_dotenv
from pydantic import BaseModel, Field

load_dotenv()

USER_AGENT = "codanics-research-agents/1.0 (https://codanics.com)"
TIMEOUT = 30


class Paper(BaseModel):
    """One normalised bibliographic record."""
    id: str = Field(description="Stable identifier, e.g. arxiv:2401.01234, pmid:38012345, doi:10.1000/xyz")
    title: str
    abstract: str = ""
    authors: list[str] = Field(default_factory=list)
    year: int | None = None
    venue: str = ""
    citations: int | None = None
    url: str = ""
    source: str = ""

    def short(self, n_abstract: int = 900) -> str:
        """Compact text block given to the LLM (keeps prompts small and citable)."""
        auth = ", ".join(self.authors[:3]) + (" et al." if len(self.authors) > 3 else "")
        cites = f" | citations: {self.citations}" if self.citations is not None else ""
        abstract = (self.abstract or "")[:n_abstract]
        return f"[{self.id}] {self.title} ({self.year}) — {auth} | {self.venue}{cites}\n{abstract}\n"


# ----------------------------------------------------------------------------- arXiv
def search_arxiv(query: str, max_results: int = 25) -> list[Paper]:
    import arxiv  # pip install arxiv

    client = arxiv.Client(page_size=min(max_results, 100), delay_seconds=3, num_retries=3)
    search = arxiv.Search(query=query, max_results=max_results,
                          sort_by=arxiv.SortCriterion.Relevance)
    papers: list[Paper] = []
    for r in client.results(search):
        papers.append(Paper(
            id=f"arxiv:{r.get_short_id()}",
            title=" ".join(r.title.split()),
            abstract=" ".join(r.summary.split()),
            authors=[a.name for a in r.authors],
            year=r.published.year if r.published else None,
            venue=(r.journal_ref or "arXiv"),
            url=r.entry_id,
            source="arxiv",
        ))
    return papers


# ------------------------------------------------------------------- Semantic Scholar
def search_semantic_scholar(query: str, max_results: int = 25, year_from: int | None = None) -> list[Paper]:
    headers = {"User-Agent": USER_AGENT}
    key = os.getenv("SEMANTIC_SCHOLAR_API_KEY")
    if key:
        headers["x-api-key"] = key
    params = {
        "query": query,
        "limit": min(max_results, 100),
        "fields": "title,abstract,year,authors,citationCount,externalIds,venue,url",
    }
    if year_from:
        params["year"] = f"{year_from}-"
    url = "https://api.semanticscholar.org/graph/v1/paper/search"
    for attempt in range(4):  # anonymous access is heavily rate-limited: back off on 429
        resp = requests.get(url, params=params, headers=headers, timeout=TIMEOUT)
        if resp.status_code != 429:
            break
        time.sleep(5 * (attempt + 1))
    resp.raise_for_status()
    papers: list[Paper] = []
    for d in resp.json().get("data", []):
        ext = d.get("externalIds") or {}
        if ext.get("DOI"):
            pid = f"doi:{ext['DOI']}"
        elif ext.get("ArXiv"):
            pid = f"arxiv:{ext['ArXiv']}"
        elif ext.get("PubMed"):
            pid = f"pmid:{ext['PubMed']}"
        else:
            pid = f"s2:{d.get('paperId')}"
        papers.append(Paper(
            id=pid,
            title=d.get("title") or "",
            abstract=d.get("abstract") or "",
            authors=[a.get("name", "") for a in d.get("authors") or []],
            year=d.get("year"),
            venue=d.get("venue") or "",
            citations=d.get("citationCount"),
            url=d.get("url") or "",
            source="semantic_scholar",
        ))
    return papers


# --------------------------------------------------------------------------- PubMed
def search_pubmed(query: str, max_results: int = 25) -> list[Paper]:
    base = "https://eutils.ncbi.nlm.nih.gov/entrez/eutils"
    common = {"tool": "codanics-research-agents", "email": os.getenv("NCBI_EMAIL", "")}
    if os.getenv("NCBI_API_KEY"):
        common["api_key"] = os.getenv("NCBI_API_KEY")

    ids = requests.get(f"{base}/esearch.fcgi", timeout=TIMEOUT, params={
        **common, "db": "pubmed", "term": query, "retmax": max_results, "retmode": "json",
        "sort": "relevance"}).json().get("esearchresult", {}).get("idlist", [])
    if not ids:
        return []

    xml = requests.get(f"{base}/efetch.fcgi", timeout=TIMEOUT, params={
        **common, "db": "pubmed", "id": ",".join(ids), "retmode": "xml"}).text
    root = ET.fromstring(xml)
    papers: list[Paper] = []
    for art in root.findall(".//PubmedArticle"):
        pmid = art.findtext(".//PMID") or ""
        title = " ".join((art.findtext(".//ArticleTitle") or "").split())
        abstract = " ".join(" ".join(t.itertext()) for t in art.findall(".//AbstractText")).strip()
        authors = []
        for a in art.findall(".//Author"):
            last, fore = a.findtext("LastName"), a.findtext("ForeName")
            if last:
                authors.append(f"{fore} {last}".strip())
        year = art.findtext(".//PubDate/Year") or art.findtext(".//ArticleDate/Year")
        venue = art.findtext(".//Journal/Title") or ""
        doi = None
        for el in art.findall(".//ArticleId"):
            if el.get("IdType") == "doi":
                doi = el.text
        papers.append(Paper(
            id=f"pmid:{pmid}", title=title, abstract=abstract, authors=authors,
            year=int(year) if year and year.isdigit() else None, venue=venue,
            url=f"https://doi.org/{doi}" if doi else f"https://pubmed.ncbi.nlm.nih.gov/{pmid}/",
            source="pubmed"))
    return papers


# ------------------------------------------------------------------------- Crossref
def search_crossref(query: str, max_results: int = 25) -> list[Paper]:
    resp = requests.get("https://api.crossref.org/works", timeout=TIMEOUT,
                        headers={"User-Agent": USER_AGENT},
                        params={"query": query, "rows": min(max_results, 100),
                                "filter": "type:journal-article",
                                "select": "DOI,title,abstract,author,issued,container-title,is-referenced-by-count,URL"})
    resp.raise_for_status()
    papers: list[Paper] = []
    for it in resp.json().get("message", {}).get("items", []):
        title = " ".join((it.get("title") or [""])[0].split())
        if not title:
            continue
        abstract = re.sub(r"<[^>]+>", " ", it.get("abstract") or "")
        abstract = " ".join(abstract.split())
        authors = [f"{a.get('given', '')} {a.get('family', '')}".strip() for a in it.get("author") or []]
        parts = (it.get("issued") or {}).get("date-parts") or [[None]]
        year = parts[0][0] if parts and parts[0] else None
        papers.append(Paper(
            id=f"doi:{it.get('DOI')}", title=title, abstract=abstract, authors=authors,
            year=year, venue=(it.get("container-title") or [""])[0],
            citations=it.get("is-referenced-by-count"), url=it.get("URL") or "", source="crossref"))
    return papers


# ---------------------------------------------------------------------- utilities
def _norm_title(t: str) -> str:
    return re.sub(r"[^a-z0-9]", "", t.lower())[:80]


def dedupe(papers: Iterable[Paper]) -> list[Paper]:
    """Merge duplicates across databases (same DOI or same normalised title)."""
    seen_ids: set[str] = set()
    seen_titles: set[str] = set()
    out: list[Paper] = []
    for p in papers:
        nt = _norm_title(p.title)
        if p.id in seen_ids or (nt and nt in seen_titles):
            continue
        seen_ids.add(p.id)
        if nt:
            seen_titles.add(nt)
        out.append(p)
    return out


SOURCES = {
    "arxiv": search_arxiv,
    "semantic_scholar": search_semantic_scholar,
    "pubmed": search_pubmed,
    "crossref": search_crossref,
}


def search_all(query: str, max_per_source: int = 20,
               sources: Iterable[str] = ("arxiv", "semantic_scholar", "pubmed", "crossref"),
               log=print) -> list[Paper]:
    """Query every requested source, tolerate failures, and return a de-duplicated list."""
    found: list[Paper] = []
    for name in sources:
        fn = SOURCES[name]
        try:
            res = fn(query, max_per_source)
            log(f"  {name:<17} {len(res):>3} records")
            found.extend(res)
        except Exception as exc:  # network / rate limit / parse errors must never kill the run
            log(f"  {name:<17} FAILED ({type(exc).__name__}: {exc})")
    papers = dedupe(found)
    log(f"  total after dedupe {len(papers):>3}")
    return papers


# -------------------------------------------------------------------- offline demo
OFFLINE_SAMPLE: list[Paper] = [
    Paper(id="demo:1", title="Deep learning for drought stress detection in wheat using RGB imagery",
          abstract="A CNN trained on RGB canopy images classifies drought stress in wheat with 92% accuracy under field conditions. Hyperspectral and genomic data were not used.",
          authors=["A. Khan", "B. Ali"], year=2023, venue="Computers and Electronics in Agriculture", citations=41, source="demo"),
    Paper(id="demo:2", title="Hyperspectral indices for early drought detection in cereals: a review",
          abstract="Reviews vegetation indices derived from hyperspectral sensors for early drought detection; highlights the lack of integration with machine learning and genotype information.",
          authors=["C. Zhang"], year=2022, venue="Remote Sensing", citations=88, source="demo"),
    Paper(id="demo:3", title="Genomic prediction of drought tolerance in wheat landraces",
          abstract="GBLUP and random forest models predict drought tolerance from SNP data across 400 landraces; phenotypic imaging data were unavailable.",
          authors=["D. Rehman", "E. Fatima"], year=2024, venue="Theoretical and Applied Genetics", citations=12, source="demo"),
    Paper(id="demo:4", title="Explainable machine learning for crop stress phenotyping",
          abstract="Applies SHAP to gradient boosting models of crop stress from multispectral UAV data; calls for multimodal fusion with genomic markers.",
          authors=["F. Ahmed"], year=2024, venue="Plant Phenomics", citations=9, source="demo"),
    Paper(id="demo:5", title="Transformer models for time-series prediction of soil moisture and crop water stress",
          abstract="A temporal transformer predicts soil moisture and canopy temperature from weather and satellite data, outperforming LSTM baselines; not validated on wheat genotypes.",
          authors=["G. Li", "H. Wang"], year=2023, venue="Agricultural Water Management", citations=27, source="demo"),
    Paper(id="demo:6", title="Multimodal data fusion in precision agriculture: challenges and opportunities",
          abstract="Surveys fusion of imaging, sensor, weather and omics data; identifies the absence of open benchmark datasets combining phenomics and genomics for drought.",
          authors=["I. Hussain"], year=2025, venue="Frontiers in Plant Science", citations=5, source="demo"),
]