Build a Pay Transparency Salary Dataset from ATS Boards

Pay ranges published on company career boards are primary-source compensation data: the employer typed the number into their own applicant tracking system. The Multi-ATS Jobs Scraper reads Workday, Greenhouse, Lever, SmartRecruiters, Workable, Ashby, Recruitee and BambooHR boards and returns salary_min, salary_max, salary_currency and salary_period exactly as configured, with the posting URL attached. Coverage is partial and uneven, and this guide shows how to measure that honestly rather than paper over it.

Why researchers should pull salary data from job postings

Compensation research has a sourcing problem. Most public salary datasets are modelled: an aggregator collects postings, fills the gaps with regression or self-reported survey data, and publishes a single blended number. That is fine for a candidate deciding whether to negotiate. It is not fine for a paper, a regulatory filing, or a board-level pay-equity review, where a reviewer will ask where the number came from and "an aggregator's estimate" is not an answer.

Statutory pay transparency has quietly fixed the sourcing problem for a growing slice of the labour market. The EU Pay Transparency Directive (EU) 2023/970 requires member states to give applicants pay information before the interview, with transposition due by 7 June 2026, and a widening set of US jurisdictions already require a range in the posting itself. Employers comply by entering a range into their applicant tracking system, which then publishes it on their public career board.

That published range is the artefact you want. It is dated, attributable to a named employer, tied to a specific requisition, and re-checkable by anyone who opens the URL. Reading it directly from the board keeps your dataset one hop from the source instead of three.

How does this compare to the alternatives?

Three routes exist to employer-published pay ranges, and they differ mainly in how much of your research time they consume before you have any data.

Writing your own collector means learning eight different ATS response shapes, each with its own pagination rules, its own salary object, and its own failure mode for a board that does not exist. A generic scraping API will fetch the HTML but hands back no schema, so you still write eight parsers and you write them again when a board changes. The actor route gives you one flat row shape across all eight systems, with the compensation fields already parsed into numbers, a currency code, and a period.

How to build a pay transparency salary dataset in 4 steps

Which career boards should I point it at first?

Start with the systems that actually carry compensation. Ashby and Lever boards publish pay bands most often; Greenhouse and Recruitee sometimes do; Workday and BambooHR essentially never do. Build your board list from the employers you want in the sample, then run a cheap probe before committing to a full collection.

from apify_client import ApifyClient

client = ApifyClient("<YOUR_APIFY_TOKEN>")

probe = client.actor("thirdwatch/multi-ats-jobs-scraper").call(run_input={
    "boardUrls": [
        "https://jobs.ashbyhq.com/ramp",
        "https://jobs.lever.co/spotify",
        "https://job-boards.greenhouse.io/stripe",
        "https://bunq.recruitee.com",
    ],
    "maxResults": 100,
    "maxResultsPerBoard": 25,
    "includeDescription": False,
})

rows = list(client.dataset(probe["defaultDatasetId"]).iterate_items())
print(len(rows), "rows")

includeDescription is off here because descriptions are irrelevant to a salary probe and cost extra requests on three of the eight systems. maxResultsPerBoard stops one large board from eating the whole sample and skewing your coverage estimate.

How do I measure salary coverage before I trust the sample?

Measure it per board, not in aggregate. A single well-configured Ashby board can make a mixed sample look healthy while every other employer in it published nothing. Compute coverage as a share of rows with a non-null salary_min, grouped by platform and company.

from collections import defaultdict

coverage = defaultdict(lambda: {"rows": 0, "with_pay": 0})

for row in rows:
    key = (row["platform"], row["company"])
    coverage[key]["rows"] += 1
    if row.get("salary_min") is not None:
        coverage[key]["with_pay"] += 1

for (platform, company), stat in sorted(coverage.items()):
    pct = 100 * stat["with_pay"] / stat["rows"]
    print(f"{platform:16} {company:20} {stat['with_pay']:4}/{stat['rows']:<4} {pct:5.1f}%")

Record this table. It is the denominator for every claim you later make, and a reviewer will ask for it. If a board shows zero percent, that is a fact about the employer's configuration, not a scraping failure.

How do I collect the full dataset once coverage looks usable?

Scale the same input up, keep filters minimal, and do the narrowing in your own code. Filtering on the board side is fast but it discards the rows that tell you what share of postings omit pay, which is itself a finding worth reporting.

run = client.actor("thirdwatch/multi-ats-jobs-scraper").call(run_input={
    "boardUrls": [
        "ashby:ramp",
        "ashby:linear",
        "lever:spotify",
        "greenhouse:stripe",
        "recruitee:bunq",
        "smartrecruiters:Visa",
        "workable:blueground",
        "workday:nvidia/wd5/NVIDIAExternalCareerSite",
    ],
    "locationFilter": "United States",
    "maxResults": 5000,
    "maxResultsPerBoard": 800,
    "includeDescription": True,
})

dataset_id = run["defaultDatasetId"]

The platform:slug shorthand is equivalent to pasting the board URL, and Workday needs all three parts (tenant, shard, site) because that is how Workday addresses a board. locationFilter is a case-insensitive substring matched against location, city, region, country and country_code, so keep it broad and refine downstream.

How do I turn mixed pay periods into one comparable column?

Convert with an explicit assumption you can defend in a methods note. Every row carries salary_period, so hourly and annual ranges are distinguishable, but nothing normalises them for you and nothing should silently.

import csv

HOURS_PER_YEAR = 2080  # 40h x 52w — state this in your methods note

def annualise(row):
    lo, hi, period = row.get("salary_min"), row.get("salary_max"), row.get("salary_period")
    if lo is None or period not in {"year", "hour", "month"}:
        return None, None
    factor = {"year": 1, "hour": HOURS_PER_YEAR, "month": 12}[period]
    return lo * factor, (hi * factor if hi is not None else None)

with open("pay_transparency_sample.csv", "w", newline="") as fh:
    writer = csv.writer(fh)
    writer.writerow(["company", "title", "location", "currency",
                     "annual_min", "annual_max", "posted_at", "job_url", "scraped_at"])
    for row in client.dataset(dataset_id).iterate_items():
        lo, hi = annualise(row)
        if lo is None:
            continue
        writer.writerow([row["company"], row["title"], row["location"],
                         row["salary_currency"], lo, hi,
                         row["posted_at"], row["job_url"], row["scraped_at"]])

Do not cross currencies in the same column either. salary_currency is an ISO code, so partition by it and convert only with a dated FX rate you cite.

How do I keep the dataset citable after the postings disappear?

Keep the provenance columns and archive the raw run. Job postings are ephemeral, so a reviewer checking your work in six months will often find a dead URL. That is survivable if you archived the untouched dataset. Export the raw run through the Apify dataset API and store it next to your cleaned file.

curl "https://api.apify.com/v2/datasets/<DATASET_ID>/items?format=json&clean=1" \
  -o raw_ats_postings_2026-07-28.json

Cite job_url for the posting, job_id plus platform for a stable key, posted_at for when the employer published it, and scraped_at for when you observed it. Those four make each row independently verifiable.

Sample output

Every posting from every ATS returns the same 28 keys, so boards from different systems concatenate without post-processing. The compensation fields are the four that matter here: salary_min and salary_max are numbers, salary_currency is an ISO code, and salary_period states what the range refers to.

[
  {
    "platform": "ashby",
    "company": "ramp",
    "job_id": "34413f8d-26bf-4bbc-8ade-eb309a0e2245",
    "title": "Security Engineer, Cloud",
    "department": "Engineering",
    "team": "Backend",
    "employment_type": "FullTime",
    "workplace_type": "hybrid",
    "is_remote": true,
    "location": "New York, NY (HQ), Remote (Canada), Remote (US)",
    "city": "New York City",
    "region": "NY",
    "country": "USA",
    "posted_at": "2026-04-07T17:12:35.753000Z",
    "salary_min": 211400,
    "salary_max": 290600,
    "salary_currency": "USD",
    "salary_period": "year",
    "job_url": "https://jobs.ashbyhq.com/ramp/34413f8d-26bf-4bbc-8ade-eb309a0e2245",
    "board_url": "https://jobs.ashbyhq.com/ramp",
    "source_query": "ashby:ramp",
    "scraped_at": "2026-07-27T22:56:42.869947Z"
  },
  {
    "platform": "workday",
    "company": "NVIDIA",
    "job_id": "JR2004417",
    "title": "Senior Site Reliability Engineer",
    "department": "Engineering",
    "employment_type": "Full time",
    "location": "US, CA, Santa Clara",
    "country_code": "US",
    "posted_at": "2026-06-30T00:00:00Z",
    "salary_min": null,
    "salary_max": null,
    "salary_currency": "",
    "salary_period": "",
    "requisition_id": "JR2004417",
    "job_url": "https://nvidia.wd5.myworkdayjobs.com/NVIDIAExternalCareerSite/job/JR2004417",
    "source_query": "workday:nvidia/wd5/NVIDIAExternalCareerSite",
    "scraped_at": "2026-07-27T22:57:11.204881Z"
  }
]

The second row is the honest case and you will see many of it: a real, current, well-formed posting with no pay range, because the employer did not publish one. Keep those rows. Dropping them at collection time destroys your ability to report coverage.

Common pitfalls

Salary coverage can be zero. On one mixed eight-board test run, department was populated on 88% of rows, description on 88%, posted date on 100%, city on 48% and salary on 0%, because none of those particular employers published ranges. Coverage is a property of employer configuration and jurisdiction, not of the collector.

Nothing is inferred. There is no estimation step anywhere in the pipeline. An empty salary_min means the board stated no number, which is exactly what makes the populated rows usable as primary evidence.

Very large Workday boards report a display-capped total. Workday commonly reports a ceiling rather than the true open count, and paging past it silently restarts at page one, so the actor stops at the reported total. Treat a very large Workday employer as a partial sample.

An unknown SmartRecruiters slug looks like an empty company. SmartRecruiters returns a successful, empty response for a company that does not exist, so a typo is indistinguishable from a firm with no open roles. Verify slugs by loading the board in a browser once.

City-level geography is thin. city was populated on under half of rows in that same test; location is the reliable field, and it is free text. Parse it yourself and report your parsing rules.

The actor handles pagination limits, retries, deduplication on platform plus board plus job ID, and treats an HTTP 200 with a non-JSON body as an unavailable board rather than a silent zero-row success.

Related use cases

The same eight-ATS collection supports several other research and operations questions. See the jobs scraping guide for the category overview, or go straight to a specific workflow:

• Audit live job postings against your ATS requisition data — find stale reqs, duplicates and mis-tagged departments before an auditor does.
• Score hiring intent signals from prospect career boards — turn open roles into a repeatable buying-intent score.
• Find which companies hire remotely by country — separate genuinely global remote roles from metro-bound ones.
• Detect closed job postings with an incremental ATS sync — keep a longitudinal panel from rotting.
• More workflows on the Thirdwatch blog.