Find Open-Access PDFs With Google Scholar Data

TL;DR — The Google Scholar Scraper returns pdfUrl — the open-access copy Scholar indexed — alongside each paper's metadata. Query a reading list, filter to records with pdfUrl, and skip the paywall chase entirely.

Why the open version is already indexed

Scholar quietly links millions of papers to free copies — arXiv preprints, repository deposits, publisher OA versions. The information exists; it's scattered one result at a time. A queryable export turns "does a free copy exist?" into a field check.

The job-to-be-done: for a reading list or citation corpus, which papers have an open PDF — and where.

How does this compare to the alternatives?

Extensions help when you're already on a page. For a hundred-paper list, you want the links in a table.

How to locate open PDFs in 4 steps

How do I check a reading list?

import os
from apify_client import ApifyClient

client = ApifyClient(os.environ["APIFY_TOKEN"])
run = client.actor("thirdwatch/google-scholar-scraper").call(
    run_input={
        "queries": [
            '"Scaling Instruction-Finetuned Language Models"',
            '"LoRA: Low-Rank Adaptation of Large Language Models"',
        ],
        "maxResults": 5,
    }
)
rows = list(client.dataset(run["defaultDatasetId"]).iterate_items())

How do I split open vs paywalled?

import pandas as pd
df = pd.DataFrame(rows).drop_duplicates("title")
open_oa = df[df.pdfUrl.notna()][["title", "pdfUrl", "venue", "year"]]
closed = df[df.pdfUrl.isna()][["title", "url", "venue"]]
print(f"{len(open_oa)} open / {len(closed)} closed")

How do I fetch the open copies?

import httpx, pathlib
for r in open_oa.to_dict("records"):
    try:
        data = httpx.get(r["pdfUrl"], timeout=30, follow_redirects=True).content
        pathlib.Path("pdfs").mkdir(exist_ok=True)
        pathlib.Path("pdfs", r["title"][:50] + ".pdf").write_bytes(data)
    except Exception:
        pass

How do I handle the closed ones?

url is the canonical page — institutional access, author copy, or interlibrary loan start there. The split itself is the useful deliverable: you know which papers need a workaround and which don't.

Sample output

{"title": "LoRA: Low-Rank Adaptation of Large Language Models",
 "url": "https://arxiv.org/abs/2106.09685",
 "authors": "E Hu, Y Shen", "venue": "ICLR", "year": "2022",
 "citedBy": 11000, "pdfUrl": "https://arxiv.org/pdf/2106.09685",
 "query": "\"LoRA: Low-Rank Adaptation\"", "position": 1}

Common pitfalls

pdfUrl absence doesn't mean closed — Scholar sometimes indexes a version without linking the PDF. Repository links rot; fetch promptly and keep the record. Publisher-hosted "free" PDFs may still carry license terms — read them before redistribution. The actor locates; rights and access rules remain yours.

Related use cases

• Scrape Google Scholar for a literature review
• Monitor new papers in your research field
• Build an arXiv research dataset
• Blog hub