Build a Jobs Aggregator from Company Career Pages (2026)

Thirdwatch's Multi-ATS Jobs Scraper is the data layer for a multi-ATS jobs aggregator — it recognises Workday, Greenhouse, Lever, SmartRecruiters, Workable, Ashby, Recruitee, and BambooHR boards from their URLs alone, returning one unified schema across all of them. Built for developers who want their own Indeed for a vertical (climate, AI, fintech) without writing eight separate scrapers.

Why build a jobs aggregator from company career pages

The case for vertical job boards is well-established. LinkedIn's 2024 talent report shows niche jobs sites consistently outperform the generalists for both engagement and conversion in defined verticals — a climate-tech jobs board converts 3–5x better than a generalist board for the same role. Building one means scraping the same set of company career pages on a schedule, normalising into a unified schema, deduping on apply URL, and serving search.

The hard part used to be the scraping layer: every ATS platform has its own HTML, JSON, or hybrid structure, and most "scrape any career page" services collapse to "scrape Greenhouse and Lever, sort of". A maintained, multi-ATS scraper with stable schema removes that floor. The job-to-be-done is concrete: ingest a list of 1,000 climate-tech (or AI, or India-tech, or whatever your vertical is) company career page URLs, refresh on a schedule, dedupe by apply URL, push into a search index. That is exactly the data shape this actor produces.

How does this compare to the alternatives?

Three options for a multi-company, multi-ATS jobs ingestion layer:

The 4–8 week DIY estimate is what an aggregator team typically burns before having a stable v1 across the major ATS platforms. The Multi-ATS Jobs Scraper actor page compresses that into a single API call. Cost-wise, the per-job rate sits well below the noise floor of what an aggregator earns in ad/affiliate revenue per visitor — the unit economics are not a concern.

How to build a jobs aggregator in 4 steps

Step 1: How do I authenticate against Apify?

Sign in at apify.com (free tier, no credit card), open Settings → Integrations, and copy your personal API token. Every example below assumes the token is in APIFY_TOKEN:

export APIFY_TOKEN="apify_api_xxxxxxxxxxxxxxxx"

Step 2: How do I batch-pull jobs from a 1,000-company list?

Apify runs have a 600-second default timeout that's plenty for hundreds of small career pages, but for thousand-company batches it's cleaner to chunk into 100-company runs. Each call passes one chunk:

import os, requests
from itertools import islice

ACTOR = "thirdwatch~multi-ats-jobs-scraper"
TOKEN = os.environ["APIFY_TOKEN"]

with open("climate-tech-careers.txt") as f:
    all_urls = [line.strip() for line in f if line.strip()]


def chunk(seq, n):
    it = iter(seq)
    while True:
        block = list(islice(it, n))
        if not block:
            break
        yield block


all_jobs = []
for batch in chunk(all_urls, 100):
    resp = requests.post(
        f"https://api.apify.com/v2/acts/{ACTOR}/run-sync-get-dataset-items",
        params={"token": TOKEN},
        json={
            "boardUrls": batch,
            "includeDescription": True,
            "maxResultsPerBoard": 500,
            "maxResults": 20000,
        },
        timeout=900,
    )
    all_jobs.extend(resp.json())
print(f"{len(all_jobs)} jobs across {len(all_urls)} companies")

A 1,000-company climate-tech aggregator at 25 jobs per company averages ~25,000 jobs per refresh.

Step 3: How do I store and dedupe in Postgres?

Postgres handles a million jobs comfortably with a single GIN index for full-text search. The schema is intentionally minimal:

CREATE TABLE jobs (
  apply_url       text PRIMARY KEY,
  title           text NOT NULL,
  company         text NOT NULL,
  department      text,
  location        text,
  employment_type text,
  description     text,
  platform        text,
  posted_at       timestamptz,
  first_seen_at   timestamptz NOT NULL DEFAULT now(),
  last_seen_at    timestamptz NOT NULL DEFAULT now()
);
CREATE INDEX jobs_search_idx ON jobs USING gin (to_tsvector('english', title || ' ' || coalesce(description, '')));
CREATE INDEX jobs_company_idx ON jobs (company);

Upsert on apply_url so re-runs don't duplicate:

import psycopg2.extras

with psycopg2.connect(...) as conn, conn.cursor() as cur:
    psycopg2.extras.execute_values(
        cur,
        """
        INSERT INTO jobs (apply_url, title, company, department, location, employment_type, description, platform, posted_at)
        VALUES %s
        ON CONFLICT (apply_url) DO UPDATE SET
          title = EXCLUDED.title,
          location = EXCLUDED.location,
          description = EXCLUDED.description,
          last_seen_at = now()
        """,
        [
            (
                j["apply_url"],
                j["title"],
                j["company"],
                j.get("department") or None,
                j.get("location") or None,
                j.get("employment_type") or None,
                j.get("description") or None,
                j.get("platform"),
                j.get("posted_at") or None,
            )
            for j in all_jobs
        ],
    )

Step 4: How do I serve search and keep the index live?

Postgres full-text covers most use cases up to about a million rows. Past that, push the dataset into a dedicated search engine — Meilisearch and Typesense both offer typo-tolerant ranked search with sub-100ms response times at this scale. Schedule the actor on Apify to run every six hours and trigger an index sync via webhook:

curl -X POST "https://api.apify.com/v2/schedules?token=$APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "climate-tech-jobs-6h",
    "cronExpression": "0 */6 * * *",
    "timezone": "UTC",
    "isEnabled": true,
    "actions": [{
      "type": "RUN_ACTOR",
      "actorId": "thirdwatch~multi-ats-jobs-scraper",
      "runInput": {"boardUrls": ["..."], "includeDescription": true, "maxResultsPerBoard": 500, "maxResults": 20000}
    }]
  }'

Add an ACTOR.RUN.SUCCEEDED webhook pointing to your ingestion service and the loop closes itself.

Sample output

A single row from the dataset looks like this (abridged — every row carries the same 28 keys, including city, country_code, salary_min/salary_max and requisition_id where the board publishes them). Five rows of this shape weigh ~25 KB on the wire and ingest into Postgres in under 50ms.

{
  "platform": "ashby",
  "company": "watershed",
  "job_id": "abc-123-789",
  "title": "Senior Backend Engineer, Carbon Markets",
  "department": "Engineering",
  "employment_type": "FullTime",
  "workplace_type": "hybrid",
  "location": "San Francisco, CA / Remote",
  "posted_at": "2026-04-21T16:02:11Z",
  "job_url": "https://jobs.ashbyhq.com/watershed/abc-123-789",
  "apply_url": "https://jobs.ashbyhq.com/watershed/abc-123-789/application",
  "description": "Watershed helps companies decarbonize... You'll own the data pipelines that ingest 100M+ emissions records...",
  "board_url": "https://jobs.ashbyhq.com/watershed"
}

platform is the field that makes a multi-ATS aggregator possible — it's set automatically from the board URL, so a Postgres WHERE platform = 'lever' filter cleanly partitions the dataset for platform-specific analytics. apply_url is universally unique because every ATS encodes a globally-unique posting ID into its URL — Lever uses a UUID, Greenhouse uses a numeric ID, Ashby uses a slug+UUID, and so on. This makes upsert-on-apply_url a robust dedupe strategy without any cross-platform collision risk.

Common pitfalls

Three things break aggregators that go to production. Stale company lists — companies pivot, get acquired, or change ATS providers, so a watchlist generated once and never updated drifts toward 20%+ dead URLs within 18 months; rebuild from your source quarterly. Description bloat — full descriptions can be 2-10 KB each, and a 100,000-job aggregator with descriptions stored verbatim in Postgres weighs 1-2 GB; consider storing descriptions in object storage (S3, R2) and keeping only an index in the DB row. Uneven field coverage — every board publishes a title, location, ID and apply URL, but department, city and pay ranges depend on how each employer configured its ATS; treat empty fields as "not published" in your UI rather than as zero, and keep companies whose careers page is not on one of the eight supported systems in a separate list for another source.

Thirdwatch's actor partitions cleanly by platform, so it's trivial to report coverage per ATS in your UI and spot boards that need attention. Out-of-band ATS changes (e.g. Greenhouse's URL pattern changing) are absorbed by Thirdwatch's parser updates, so the aggregator keeps running while the underlying world shifts.

Related use cases

• Scrape Greenhouse jobs for ATS enrichment
• Scrape Lever jobs for a recruiter sourcing pipeline
• Track startup hiring velocity with career site data
• The complete guide to scraping job boards
• All Thirdwatch use-case guides