Analyze arXiv Papers With Python
Analyze arXiv paper metadata in pandas, including submission trends, categories, authors, revisions, and abstract text with correct data grain.
Load the export from the arXiv Papers Scraper and establish paper grain:
import pandas as pd
papers = pd.read_json("dataset_items.json").drop_duplicates("arxiv_id")
papers["published"] = pd.to_datetime(papers["published"], utc=True)
categories = papers[["arxiv_id", "categories"]].explode("categories")
print(categories["categories"].value_counts().head(20))Use the paper table for counts and the exploded table for category memberships. Derive a versionless ID if multiple versions are collected, but keep the source ID. For abstract analysis, inspect preprocessing samples and report the exact query set.
Submission volume reflects arXiv usage and the chosen scope. It does not measure peer-reviewed output or research quality.
Frequently asked questions
Should category arrays be exploded?
Yes for category analysis, but keep a paper table for unique-paper counts.
Which date represents a revision?
Use updated for revision timing and published for the initial submission timestamp.
Related
100 free credits, no credit card.
About 30 real searches. Add the MCP to Claude or Cursor in two minutes.