Build an arXiv Research Dataset
Combine transparent arXiv queries into a version-aware paper dataset for landscape reviews, topic analysis, and research discovery.
Define the research scope before collection: categories, title or abstract phrases, exclusions, dates implied by the sort window, and maximum results. Run each expression separately with the arXiv Papers Scraper and attach a query label to every row.
Preserve an immutable source table. Build derived paper, author, and category tables from it. Use the base arXiv ID for paper-level counts while retaining version, published date, and updated date. Do not infer peer-review status from a DOI or journal-reference field alone.
When queries overlap, report both unique papers and query matches. The overlap itself can be analytically useful, but only if it is not hidden by premature deduplication.
Frequently asked questions
Is arXiv a complete academic literature database?
No. Coverage varies by discipline and it primarily contains preprints and author submissions.
How should duplicate versions be represented?
Keep the source version and derive a base ID for paper-level deduplication.