Build a Literature Dataset from Crossref Metadata

A literature dataset for landscape analysis needs stable identifiers, documented discovery rules, and explicit missing-data handling. The Crossref DOI Scraper supplies structured member-deposited metadata that can become one layer of that dataset.

Store the search strategy

Choose queries, dates, work types, and caps before collection. Run themes separately if the analysis needs a query-to-work relationship.

{
  "queries": ["software supply chain security"],
  "fromPublishedDate": "2020-01-01",
  "workType": "journal-article",
  "maxResultsPerQuery": 500
}

Archive input JSON, Actor run ID, collection time, and raw results. Normalize DOI for comparisons, while preserving titles, authors, affiliations, container titles, publisher, dates, identifiers, licenses, counts, and canonical URLs.

Model metadata honestly

Use a work table keyed by DOI, an author table when person-level analysis is needed, and bridge tables for authorship and affiliations. Author strings are not universal identities; prefer ORCID where deposited, while recognizing it is often absent.

Do not choose one publication date silently. Store online, print, and general published dates separately, then document which date an analysis uses. Preserve nulls for missing language, ISSN, ISBN, license, or affiliation.

Crossref search relevance and result caps affect coverage. A dataset that hits the cap is not complete. Supplement it with other databases when the research question requires broader indexing or full text.

For refreshes, compare DOI keys and Crossref indexed timestamps across successful snapshots. Member corrections can alter metadata without creating a new work. Retain raw versions so transformed tables and published analyses can be reproduced later.