Build a Research Reading List From Scholar Data

TL;DR — The Google Scholar Scraper turns a topic's queries into a structured paper list — citations, venues, snippets, PDF links. Dedupe across queries, sort by impact, and your next reading list is a ranked table, not 40 open tabs.

Why reading lists fail without structure

Topic exploration generates tabs: forty Scholar results, no ranking rationale, no dedup. The list that survives is the one that's a table — every paper scored by citations, filtered by year, with a snippet explaining why it matched.

The job-to-be-done is a merge of queries into one prioritized list — the papers multiple search angles agree on rising first.

How does this compare to the alternatives?

Reference managers organize papers you've chosen. The actor builds the candidate set you choose from.

How to build the list in 4 steps

How do I cover a topic from multiple angles?

import os
from apify_client import ApifyClient

client = ApifyClient(os.environ["APIFY_TOKEN"])
run = client.actor("thirdwatch/google-scholar-scraper").call(
    run_input={
        "queries": [
            "vector database indexing",
            "approximate nearest neighbor search",
            "embedding retrieval at scale",
        ],
        "fromYear": 2022,
        "maxResults": 30,
    }
)
rows = list(client.dataset(run["defaultDatasetId"]).iterate_items())

How do I dedupe and rank?

import pandas as pd
df = pd.DataFrame(rows)
counts = df.groupby("title")["query"].nunique()
df["hits"] = df.title.map(counts)
top = (df.drop_duplicates("title")
       .sort_values(["hits", "citedBy"], ascending=False))
print(top[["title", "venue", "year", "citedBy", "hits"]].head(15))

Papers surfaced by multiple queries (hits > 1) are the topic's center of gravity — read those first.

How do I attach reading context?

top["pdf"] = top["pdfUrl"]
top[["title", "authors", "year", "citedBy", "pdf", "snippet"]].to_csv(
    "reading_list.csv", index=False)

snippet carries Scholar's matched excerpt — usually enough to triage without opening the paper.

How do I keep it current?

Monthly reruns with fromYear bumped catch new entrants; diff on title to append rather than rebuild.

Sample output

{"title": "Billion-scale similarity search with GPUs",
 "url": "https://arxiv.org/abs/1702.08734",
 "authors": "J Johnson, M Douze, H Jégou", "venue": "IEEE TBD",
 "year": "2019", "snippet": "…GPU-accelerated nearest neighbor…",
 "citedBy": 3900, "pdfUrl": "https://arxiv.org/pdf/1702.08734",
 "query": "approximate nearest neighbor search", "position": 3}

Common pitfalls

Citation counts favor older work — weight year when the field moves fast. Title dedupe misses editions; a fuzzy match on title+first author catches them. snippet is query-biased — it shows the match, not the abstract. The actor supplies the candidate set; the reading remains your craft.

Related use cases

• Scrape Google Scholar for a literature review
• Find open-access PDFs with Google Scholar data
• Monitor new papers in your research field
• Blog hub