Compare FEC Candidate Campaign Finance Totals with Apify

TL;DR: The FEC Campaign Finance Scraper turns public OpenFEC records into a repeatable fec candidate finance comparison dataset for political researchers. It accepts focused discovery inputs, saves structured evidence, deduplicates records, and exposes explicit result limits. Use it when a browser export or a one-off script would make comparisons difficult to reproduce. Keep dated snapshots, validate row counts, and join additional security or ownership evidence separately instead of stretching public-source data beyond what it proves.

Why scrape OpenFEC for finance comparison?

FEC candidate finance comparison replaces manual catalog checks with comparable, time-stamped evidence. A source website answers one question at a time, while political researchers usually need to compare a cohort, repeat the same query next week, and explain why an option was shortlisted. The useful unit is therefore not a screenshot; it is a stable table with provenance.

The OpenFEC record contributes candidate identity, office, state, party, election years, principal committee identity, totals availability, cycle coverage, aggregate receipts, aggregate contributions, spending, cash, and debt. Those fields support discovery, triage, trend analysis, and watchlists without claiming that popularity proves security or product quality. The official OpenFEC API documentation describes the source and its public access model. Apify's Tasks documentation explains how tested inputs become repeatable scheduled jobs.

This workflow is especially useful when the decision has a defined population and cadence: defined research cohorts, regulated records, funded projects, or identities that require review. Start with a narrow question, retain the raw evidence, and calculate rankings downstream. That separation keeps collection reproducible and analysis revisable.

How does this compare to the alternatives?

A purpose-built Actor gives finance comparison a smaller operational surface than maintaining a custom collector. The source itself remains authoritative; the Actor standardizes extraction, limits, deduplication, and delivery.

Method	Commercial model	Reliability	Setup time	Ongoing maintenance
DIY Python script	Engineering time plus hosting	Depends on local retry and schema handling	Hours to days	Owned by your team
Generic scraping API	Usage or subscription	Page-oriented and source-dependent	Hours	Selectors and pagination remain yours
Thirdwatch Actor	Pay per saved result	Source-specific validation and retries	Minutes	Managed listing and schema updates

The DIY route can be appropriate for a deeply customized internal system. A generic API helps when rendered pages are the only source. For public public-source data, the actor page offers a direct path with inputs that match the actual collection modes and output shaped for datasets.

How to run fec candidate finance comparison in 5 steps

What question should the dataset answer?

The first step is to write one decision question with a defined population and review cadence. Examples include identifying maintained options, monitoring a production watchlist, or comparing activity within one category. Avoid a query like "all software," because a large result set makes neither the inclusion rule nor the refresh strategy clear.

Write down the audience, expected result range, fields required for the decision, and what happens when a value changes. That short contract prevents silent scope expansion. For market research, discovery queries are appropriate. For governance, exact names are usually stronger because the cohort should not drift between runs.

Which Actor input should I use?

Use discovery inputs for broad research and exact-name inputs for fixed watchlists. The Actor schema exposes only supported fields, and the default limits keep the first run small. This example combines the modes supported by this Actor:

{"queries":["Smith"],"candidateIds":[],"cycle":2026,"office":"S","includeTotals":true,"maxResultsPerQuery":25,"maxResults":25}

Keep separate Tasks when two queries represent different business questions. Combined runs deduplicate by candidateId, which is useful for a canonical table but can hide query membership. If membership matters, store a bridge table containing query, key, run ID, and collection time.

How do I run it through the Apify API?

The Apify client can start the Actor and return its dataset in one reproducible script. Put the token in an environment variable rather than source control.

import os
from apify_client import ApifyClient

client = ApifyClient(os.environ["APIFY_TOKEN"])
run = client.actor("thirdwatch/fec-campaign-finance-scraper").call(
    run_input={
        "queries": ["Smith"],
        "candidateIds": [],
        "cycle": 2026,
        "office": "S",
        "includeTotals": true,
        "maxResultsPerQuery": 25,
        "maxResults": 25,
    }
)
items = list(client.dataset(run["defaultDatasetId"]).iterate_items())
print(f"saved {len(items)} records from run {run['id']}")

For production automation, set an explicit timeout, log the run ID, and fail the downstream load if the Actor run is not successful. The Apify API documentation covers run and dataset endpoints for non-Python clients.

How should I validate the returned records?

Validation should check identity, evidence URLs, counts, and the fields required by the decision. Do not reject a row merely because an optional description or popularity metric is null. Public datasets contain records of different ages and publishing practices, so optional-field completeness is not uniform.

required = ["candidateId", "sourceUrl", "source"]
bad = [row for row in items if any(not row.get(field) for field in required)]
if bad:
    raise ValueError(f"{len(bad)} rows failed identity validation")
if len(items) == 0:
    raise ValueError("The run returned no evidence; keep the previous snapshot")

Add range checks for counts, parse timestamps into UTC, and compare current volume with the previous successful run. A sudden collapse often indicates a changed query, upstream outage, or rate limit,not a real market event.

How do I schedule and store the snapshots?

A saved Task turns the tested input into a stable collection contract. Choose a cadence that matches the signal: daily for release watchlists, weekly for operational portfolios, and monthly for market maps. Write each successful run to an immutable dated partition before updating a current-state table.

Use candidateId as the natural key, collected_at as snapshot time, and the source URL as evidence. Calculate changes after ingestion: new records, removed records, version transitions, activity changes, and rank movement. Alert only on changes tied to a decision; indiscriminate alerts teach teams to ignore the feed.

What does the OpenFEC output look like?

The output preserves a canonical identity and the public metadata needed for finance comparison. A representative record looks like this:

{"candidateId":"S8FL00216","name":"BARTLETT, HAMILTON ALLEN SMITH","office":"Senate","party":"REP","totalsStatus":"available","receipts":3852434.85,"disbursements":831532.74,"totalsCycle":2026,"source":"Federal Election Commission public data"}

The candidateId field is the durable join key. The sourceUrl is the human-review path, while source identifies the upstream system. Dates, classifications, amounts, status fields, and source identifiers are snapshot evidence. Keep the raw row even if a downstream model uses only a subset; reprocessing is cheaper and more defensible than recollecting historical state.

Common pitfalls in fec candidate finance comparison

The most common failure is treating public-source data as a complete risk or quality verdict. Federal law restricts sale or commercial use of individual contributor information. This Actor intentionally returns no donor rows, names, employers, occupations, locations, or contact details.

Other pitfalls include mixing discovery and watchlist rows without recording mode, overwriting the previous good snapshot after an empty run, comparing rank across different query caps, and alerting on mutable tag names without retaining immutable version or digest evidence. Rate limits also make aggressive concurrency counterproductive.

Use explicit limits, stable queries, UTC collection times, and immutable snapshots. Join review decisions and external evidence in separate, attributed tables. The Thirdwatch Actor handles bounded requests, retries, deduplication, and public-source normalization; your downstream model remains responsible for the business decision.

Related use cases

These adjacent workflows extend the same evidence-first collection pattern. Continue with:

• Monitor Fec Candidate Receipts And Spending
• Build Fec Candidate Committee Dataset
• Analyze Campaign Cash And Debt By Cycle
• Build an npm package intelligence dataset
• Explore all Thirdwatch data workflows
• Read the business-data scraping guide

Source boundary and responsible use

The Actor never calls individual-contributor endpoints and removes treasurer, agent, phone, and address fields. With the shared DEMO_KEY, candidate discovery uses the official FEC cycle candidate-master file; supplied keys use live OpenFEC search. Totals depend on filing coverage and amendments. Each row reports totalsStatus, and a private OpenFEC key supports steadier enrichment at scale.

Federal law restricts sale or commercial use of individual contributor information. This Actor intentionally returns no donor rows, names, employers, occupations, locations, or contact details. The Actor saves a source URL and input query with every record so reviewers can trace a finding. Keep the unmodified dataset alongside any derived score, and label internal judgments as analysis rather than source facts.

Start with a research question, not a download target

For FEC candidate finance comparison, write the question before choosing a result limit. Name the population, time window, and comparison unit. A useful question can be answered by a table and has an explicit stopping point. "Collect everything" is neither testable nor cheap.

Run a ten-row sample first. Check the canonical identifier, source link, dates, and fields used in the decision. Then inspect two records directly on OpenFEC. If terminology differs from the source documentation, fix the interpretation before scaling. Keep the sample dataset and the final input beside the analysis so another researcher can reproduce the choice of records.

The final report should separate observations from conclusions. Source fields belong in the evidence table. Thresholds, classifications, and reviewer notes belong in derived columns with their rule version. This matters because the same public record can support several analyses without changing what the upstream source actually published.

Design the discovery funnel

Discovery works best as a funnel, not a single oversized query. Begin with three to five phrases that describe the capability from different angles: the technical category, the job a team needs done, and a common implementation term. Save query membership beside each returned identity. Overlap between phrases is useful evidence because it shows which records appear across several ways of expressing the same need.

In the second pass, remove records that fail an explicit eligibility rule. The rule might require a source repository, recent activity, a supported runtime, an approved license, or a minimum adoption signal. Keep the exclusion reason in a separate review table. A reproducible exclusion trail is more valuable than silently deleting rows, especially when a stakeholder later asks why a familiar option is absent.

The third pass is qualitative. Select a manageable shortlist and review the canonical source pages. Descriptions, counts, and tags accelerate triage but cannot prove maintenance quality, security posture, or organizational fit. Record the review date and reviewer, then keep the collector output unchanged. This preserves the distinction between upstream facts and internal judgment.

Measure the funnel with four numbers: discovered identities, eligible identities, reviewed identities, and approved identities. Compare those counts between runs using the same queries and limits. If the discovered population changes sharply, inspect upstream ranking and pagination before concluding that the market changed. If only approval changes, the cause is probably an internal policy or evidence update.

For ongoing research, keep discovery and monitoring as separate Tasks. Discovery can run monthly with broader phrases; the approved shortlist can run daily or weekly through exact identifiers. The result is a market map that can evolve without making a production watchlist drift unexpectedly. That division also keeps alert volume proportional to decisions rather than to catalog noise.

Discovery metrics that expose bad assumptions

Track query yield, cross-query overlap, eligibility rate, and evidence completeness before tracking a winner. Low yield can mean the vocabulary is wrong; high overlap can mean the phrases are redundant; a low eligibility rate can mean the catalog does not match the requirement. Sample records from both the accepted and rejected groups. Review the sample against canonical pages, then revise one rule at a time so the effect remains measurable.

Do not compare rank movement unless the query, sort, limits, and collection mode are unchanged. Instead, calculate new identities, disappeared identities, and material field changes between matched snapshots. Publish the exact input with the report. A reader should be able to distinguish a changed market from a changed search plan without relying on the analyst's memory.