Detect Closed Job Postings with an Incremental ATS Sync

Thirdwatch's Multi-ATS Jobs Scraper reads company career boards across Workday, Greenhouse, Lever, SmartRecruiters, Workable, Ashby, Recruitee and BambooHR into one flat schema. Run it on a schedule, diff each run against the last on platform + company_slug + job_id, and you get three signals a one-shot scrape never gives you: which reqs are new, which disappeared, and how long each one stayed open. This is the maintenance half of a jobs feed, written for the engineers who own the second run onward.

Why disappearance is the signal your job index is missing

A job posting that vanishes from a company career board is data, not an absence of data. Employers do not publish a "filled" flag; they take the posting down. That deletion is the closest public proxy you get to a hiring outcome, and the gap between posted_at and the run where the posting stopped appearing is a bounded estimate of time-to-fill. SHRM's talent acquisition benchmarking research puts the average time to fill a role in the mid-40s of days, which means a daily sync brackets almost every closure to within 24 hours of its true date.

The alternative — never diffing — is worse than it sounds. An index that only ever inserts is an index that rots. After a quarter of daily ingests without a closure signal, a meaningful share of your rows are roles filled weeks ago, and every downstream consumer inherits that error: candidates applying into dead links, an SDR citing a req that closed in May, a market report counting the same opening four times.

The job to be done is narrow. Snapshot every board you track, compare it to the previous snapshot, classify each key as opened, still open or closed, and — critically — refuse to classify anything when the read itself looks suspect.

How does this compare to the alternatives?

Three ways to get a closure signal on company career boards.

The distinction that matters for closure detection is not price, it is source. Aggregators derive from the same boards you could read directly and retain listings long after the employer pulled them, so "missing from the aggregator" is uncorrelated with a real close. Reading the company board directly makes disappearance mean something. If you are still building the first ingest, start with building a jobs aggregator from company career pages and come back here for the sync loop.

How to detect closed job postings in 4 steps

How do I run a baseline sync of every board I track?

Sync the full board, not a filtered slice. Turn descriptions off — a diff only needs identity fields, and descriptions cost an extra request per posting on three of the eight systems.

from apify_client import ApifyClient
from datetime import datetime, timezone

client = ApifyClient("<YOUR_APIFY_TOKEN>")

BOARDS = [
    "https://job-boards.greenhouse.io/stripe",
    "https://jobs.lever.co/spotify",
    "https://jobs.ashbyhq.com/linear",
    "smartrecruiters:Visa",
    "recruitee:bunq",
    "workday:nvidia/wd5/NVIDIAExternalCareerSite",
]

run_at = datetime.now(timezone.utc)

run = client.actor("thirdwatch/multi-ats-jobs-scraper").call(run_input={
    "boardUrls": BOARDS,
    "includeDescription": False,     # identity-only sync: fewer requests, faster
    "searchText": "",                # never keyword-filter a diff source
    "maxResults": 20000,
    "maxResultsPerBoard": 5000,      # one huge board can't eat the whole budget
})

rows = list(client.dataset(run["defaultDatasetId"]).iterate_items())

maxResultsPerBoard matters more here than in a one-off pull. Without it, a single 4,000-row Workday tenant can eat the budget and starve the other boards — and a starved board looks exactly like an emptied one to a naive diff.

How do I diff this run against the last one?

Key on platform, company_slug and job_id. That triple is what the actor itself deduplicates on within a run, and it survives title edits and URL slug changes.

def key(row):
    return (row["platform"], row["company_slug"], row["job_id"])

current = {key(r): r for r in rows}
previous = load_snapshot()          # same shape, written by the last run

opened     = [current[k]  for k in current.keys()  - previous.keys()]
vanished   = [previous[k] for k in previous.keys() - current.keys()]
still_open = current.keys() & previous.keys()

print(f"{len(opened)} new, {len(vanished)} vanished, {len(still_open)} unchanged")

Note the variable name: vanished, not closed. Nothing is a closure until it survives the next step.

How do I avoid recording a false closure when a board read goes wrong?

Gate every closure on the health of the board it came from, never on the individual row. Three source behaviours produce a clean, successful, wrong-looking run: a very large Workday board reports a display-capped total and can hand back a different subset between runs; SmartRecruiters answers an unknown slug with a normal empty response, so a typo is indistinguishable from a company with nothing open; and zero results is a documented, non-error outcome everywhere.

from collections import Counter

def board_counts(records):
    return Counter((r["platform"], r["company_slug"]) for r in records)

now, before = board_counts(rows), board_counts(previous.values())

trusted = set()
for board, prior_count in before.items():
    latest = now.get(board, 0)
    if latest == 0 and prior_count > 0:
        continue                                   # unread, not emptied
    if prior_count >= 20 and latest < prior_count * 0.5:
        continue                                   # >50% collapse in one cycle
    trusted.add(board)

closed = [r for r in vanished if (r["platform"], r["company_slug"]) in trusted]
skipped = {b for b in before if b not in trusted}

A board that fails the gate is not an error to alert on — it is a board whose closures you defer to the next run. Two consecutive skips on the same board is what deserves a page, because that usually means a renamed board or a bad slug rather than a flaky read.

How do I turn a disappearance into a time-to-fill estimate?

You know the posting was live at the previous run and gone at this one, so the true close date sits inside that window. Report the bounds rather than a single fabricated number.

def parse(ts):
    return datetime.fromisoformat(ts.replace("Z", "+00:00")) if ts else None

for r in closed:
    posted = parse(r.get("posted_at"))
    r["closed_after"]  = previous_run_at.isoformat()
    r["closed_before"] = run_at.isoformat()
    r["days_open_min"] = (previous_run_at - posted).days if posted else None
    r["days_open_max"] = (run_at - posted).days if posted else None

posted_at came back populated on every row of a mixed eight-board test run, which is the coverage this estimate depends on. updated_at is far patchier and is a repost signal at best.

How do I keep sync state between scheduled runs?

Write the snapshot somewhere that outlives the run. Apify's default key-value store is scoped to a single run, so use a named store (or your own database) for anything cross-run.

store_id = client.key_value_stores().get_or_create(name="ats-sync-state")["id"]
kv = client.key_value_store(store_id)

kv.set_record("last-sync", {
    "run_at": run_at.isoformat(),
    "rows": {
        "|".join(k): {
            "title": v["title"],
            "posted_at": v.get("posted_at"),
            "job_url": v["job_url"],
            "board_url": v["board_url"],
        }
        for k, v in current.items()
    },
})

Then put it on a schedule at a fixed hour so your windows are even-width. The key-value store docs cover retention if you want a rolling history of snapshots rather than only the last one.

Sample output

Each run returns the same 28 keys regardless of which ATS a posting came from, which is what makes a cross-board diff possible at all. These are the fields the sync loop above actually reads.

[
  {
    "platform": "ashby",
    "company": "ramp",
    "company_slug": "ramp",
    "job_id": "34413f8d-26bf-4bbc-8ade-eb309a0e2245",
    "title": "Security Engineer, Cloud",
    "department": "Engineering",
    "team": "Backend",
    "employment_type": "FullTime",
    "workplace_type": "hybrid",
    "is_remote": true,
    "location": "New York, NY (HQ), Remote (Canada), Remote (US)",
    "city": "New York City",
    "region": "NY",
    "country": "USA",
    "country_code": "",
    "posted_at": "2026-04-07T17:12:35.753000Z",
    "updated_at": "",
    "salary_min": 211400,
    "salary_max": 290600,
    "salary_currency": "USD",
    "salary_period": "year",
    "requisition_id": "",
    "job_url": "https://jobs.ashbyhq.com/ramp/34413f8d-26bf-4bbc-8ade-eb309a0e2245",
    "apply_url": "https://jobs.ashbyhq.com/ramp/34413f8d-26bf-4bbc-8ade-eb309a0e2245/application",
    "description": "",
    "board_url": "https://jobs.ashbyhq.com/ramp",
    "source_query": "ashby:ramp",
    "scraped_at": "2026-07-27T22:56:42.869947Z"
  },
  {
    "platform": "greenhouse",
    "company": "Stripe",
    "company_slug": "stripe",
    "job_id": "6217043",
    "title": "Staff Data Engineer, Risk",
    "department": "Engineering",
    "team": "",
    "employment_type": "",
    "workplace_type": "onsite",
    "is_remote": null,
    "location": "Seattle, Washington, United States",
    "city": "Seattle",
    "region": "Washington",
    "country": "United States",
    "country_code": "US",
    "posted_at": "2026-06-19T00:00:00Z",
    "updated_at": "2026-07-02T00:00:00Z",
    "salary_min": null,
    "salary_max": null,
    "salary_currency": "",
    "salary_period": "",
    "requisition_id": "",
    "job_url": "https://job-boards.greenhouse.io/stripe/jobs/6217043",
    "apply_url": "https://job-boards.greenhouse.io/stripe/jobs/6217043#app",
    "description": "",
    "board_url": "https://job-boards.greenhouse.io/stripe",
    "source_query": "https://job-boards.greenhouse.io/stripe",
    "scraped_at": "2026-07-27T22:56:44.113209Z"
  }
]

The diff needs six of them: platform, company_slug and job_id form the key; board_url and source_query tell you which input entry produced the row when a board needs investigating; posted_at anchors the duration estimate. is_remote is null rather than false on boards that publish no remote flag, so never treat it as a boolean without checking, and salary_min / salary_max are populated only where the employer published a range.

Common pitfalls

Workday's display-capped total. Very large Workday boards report a ceiling — commonly 2000 — instead of the real count, and paging past it silently restarts at page one. The actor stops at the reported total, so a board near that ceiling can legitimately return a different subset run to run. Gate it behind the row-count check above.

SmartRecruiters typos look like empty boards. An unknown company slug returns a successful, empty response rather than a not-found. The run log flags the ambiguity but cannot resolve it. If a SmartRecruiters board goes from 40 rows to zero, assume a slug problem before you assume 40 closures.

Zero results is not an error. An empty board, an over-narrow filter or a keyword nothing matches all finish cleanly with a warning. Your pipeline must treat a clean zero as "do not diff", not as "everything closed".

Field coverage varies by ATS. On a mixed eight-board test run, posted_at was populated on every row and department on most, but city on under half and salary on none — because those particular employers published no ranges. Build the diff on the fields that are always there.

Reposts masquerade as churn. An edited, re-published req can arrive with a new job_id, showing up as one close plus one open on the same day. Match closes against opens on the same board with a similar title before reporting either.

Thirdwatch handles parsing, pagination limits and retries for all eight systems, and treats an HTTP 200 with a non-JSON body as an unavailable board rather than a success — so the zero-row runs you do see are real, and worth gating on.

Related use cases

• Audit live job postings against your ATS requisition data — the recruiting-ops view of the same board reads.
• Build a pay transparency salary dataset from ATS boards — for when the published ranges are the point.
• Score hiring intent signals from prospect career boards — closures and openings as a GTM trigger.
• Find which companies hire remotely by country — reading is_remote, country_code and location fields properly.
• Build a jobs aggregator from company career pages — the greenfield build this post picks up from.
• Guide to scraping job boards — the category pillar, plus everything else on the Thirdwatch blog.