The Complete Guide to Scraping Compliance & Risk Data (2026)

Thirdwatch covers 14 regulatory and risk sources spanning company registries, sanctions and debarment lists, securities filings, vulnerability catalogs and product recalls. This guide is the decision tree for assembling a compliance data set — KYB onboarding, sanctions screening, vendor and supplier risk, credit risk, filings research, and recall monitoring.

The compliance data landscape

Compliance data has the opposite problem to most scraping categories. The sources are official, free and deliberately public — and almost unusable in that state. Registries paginate badly, sanctions lists ship as XML with regime-specific field sets, filings sit behind identifiers you have to resolve first, and recall notices arrive as prose. The work is never getting the page; it is turning fifteen incompatible official formats into one row per entity that a risk engine can query.

The refresh problem is what actually bites teams. Designations take legal effect the moment they are published: the U.S. Treasury's OFAC recent actions feed changes several times a month, and a screening decision made against a stale copy is a control failure regardless of intent. On the security side the National Vulnerability Database published well over 30,000 new CVE records in 2024, which no team triages by hand.

For a KYB or onboarding function, the right answer is a registry source plus a tax or status source plus watchlists. For third-party risk, add insolvency and debarment. For security and product teams, vulnerability and recall feeds stand alone.

Compare Thirdwatch compliance and risk sources

Source	What it covers	Availability	Best for
SEC EDGAR	US filings: 10-K, 10-Q, 8-K, XBRL financials	On Apify Store	Public-company diligence
MCA India	Indian company master data, directors, CIN, capital	On Apify Store	India KYB and director checks
GST Verification	GSTIN validation, legal name, status, jurisdiction	On Apify Store	India vendor onboarding
IBBI	Indian insolvency (CIRP), liquidation, professionals	On Apify Store	India credit and counterparty risk
OFAC sanctions	US Treasury SDN and consolidated lists, programs	On request	US sanctions screening
UK sanctions list	FCDO designations, regimes, measures, aliases	On request	UK sanctions screening
UN sanctions list	Security Council consolidated list, regimes	On request	Global baseline screening
World Bank debarment	Ineligible firms and individuals, reasons, dates	On request	Procurement and supplier vetting
GLEIF LEI	Legal Entity Identifiers, jurisdiction, renewal status	On request	Counterparty identity resolution
India government tenders	Central procurement notices, deadlines, EMD, value	On request	Public-procurement compliance
CISA KEV	Vulnerabilities confirmed exploited, remediation dates	On request	Patch prioritisation
NVD CVE	CVE records, CVSS, CWE, affected products	On request	Vulnerability intelligence
FDA recalls	Food, drug and device enforcement records	On request	Product safety monitoring
CPSC recalls	US consumer-product recalls, hazards, UPCs, remedies	On request	Retail and marketplace safety

Two further FDA sources cover drug adverse-event reports and structured product labels, including boxed warnings and dosage sections. Everything marked "on request" is provisioned directly rather than listed for self-serve use; the extraction and output schema are the same either way.

Decision tree

"I'm onboarding Indian vendors and need a KYB check." GST Verification plus MCA India plus IBBI, in that order. GST validates the GSTIN format and checksum before you spend a lookup, then returns legal name, trade name, registration status and jurisdiction. MCA resolves the CIN to company master data, directors with DIN, registered address and capital. IBBI tells you whether the counterparty is in an insolvency process — the single most decision-relevant field in Indian vendor risk, and the one most onboarding flows omit.

"I'm screening counterparties against sanctions." UN as the global baseline, OFAC for US exposure, UK FCDO for UK exposure, World Bank debarment where procurement or development finance is involved. Screen on aliases as well as primary names; every one of these lists carries alias fields precisely because designated parties do not use their registered names.

"I'm doing diligence on a US public company." SEC EDGAR with filingType set to 10-K and includeFinancials enabled returns revenue, net income, total assets, EPS and multi-year history in the same record as the filing metadata. Add 8-K monitoring for event-driven signals — material agreements, auditor changes, executive departures.

"I'm resolving who a counterparty actually is across systems." GLEIF LEI records are the cleanest identity spine available, because the LEI is jurisdiction-independent and carries registration and renewal status. Use it as the join key between your CRM, your registry lookups and your screening results, rather than inventing an internal entity ID.

"I'm running vulnerability management." CISA KEV first, because it is short and each entry carries a remediation deadline, then the NVD for breadth on the products in your inventory. Filtering the NVD by severity alone produces an unworkable queue; filtering by "in KEV" plus "matches a CPE in our asset inventory" produces a list a team can actually clear.

"I'm monitoring product safety for a marketplace or retailer." CPSC for consumer products, FDA for food, drug and device enforcement. Match on UPC and brand where present, and treat Class I FDA recalls as a separate alerting tier — those are the ones with a reasonable probability of serious harm.

"I'm tracking public procurement." India government tenders for central-ministry notices with deadlines, EMD amounts and tender value. This is as much a sales-pipeline source as a compliance one; see the business data pillar for the lead-routing side.

Cross-source recipe: an India vendor due-diligence check

import os, requests, pandas as pd

TOKEN = os.environ["APIFY_TOKEN"]


def run(actor, payload, timeout=3600):
    r = requests.post(
        f"https://api.apify.com/v2/acts/{actor}/run-sync-get-dataset-items",
        params={"token": TOKEN},
        json=payload,
        timeout=timeout,
    )
    return r.json()


GSTINS = ["27AADCB2230M1ZT", "29AAACR5055K1ZL"]

gst = pd.DataFrame(
    run("thirdwatch~gst-verification-scraper", {"queries": GSTINS, "maxResults": 100})
)

# Only pay for a registry lookup on GSTINs that actually verified.
verified = gst[gst.verification_status == "verified"]
names = verified.business_name.dropna().tolist()

mca = pd.DataFrame(
    run(
        "thirdwatch~mca-india-scraper",
        {"queries": names, "includeDirectors": True, "maxResults": len(names)},
    )
)

ibbi = pd.DataFrame(
    run(
        "thirdwatch~ibbi-scraper",
        {
            "queries": names,
            "searchType": "corporate_debtor",
            "status": "all",
            "maxResults": 25,
        },
    )
)


def norm(s):
    return s.str.upper().str.replace(r"[^A-Z0-9]", "", regex=True)


verified = verified.assign(key=norm(verified.business_name))
mca = mca.assign(key=norm(mca.company_name))
ibbi = ibbi.assign(key=norm(ibbi.company_name))

view = verified.merge(
    mca[["key", "cin", "status", "incorporation_date", "directors"]],
    on="key",
    how="left",
    suffixes=("_gst", "_mca"),
)
view["insolvency_flag"] = view.key.isin(set(ibbi.key))
view["clear_to_onboard"] = (
    (view.status_gst == "Active")
    & (view.status_mca == "Active")
    & ~view.insolvency_flag
)

print(view[["gstin", "business_name", "cin", "insolvency_flag", "clear_to_onboard"]])

Note the ordering. Format and checksum validation happens before any network call, GST status gates the registry lookup, and insolvency is checked only against names that survived both. A naive pipeline that fans all three sources out in parallel over the raw input list does several times the work for the same answer.

All use-case guides for compliance scrapers

SEC EDGAR

• Scrape SEC EDGAR filings for financial research
• Extract SEC EDGAR financials for stock screening
• Monitor SEC 10-K filings for competitive intel
• Track SEC 8-K filings for event-driven research
• Build an SEC filing database for investment analysis

India registries

• Scrape MCA India company data for due diligence
• Look up MCA directors for KYC compliance
• Verify Indian vendors with MCA company data
• Verify Indian GST numbers at scale
• Monitor GST status changes for vendor due diligence
• Screen Indian vendors for insolvency risk
• Scrape IBBI insolvency cases for credit risk
• Monitor IBBI CIRP cases for legal compliance

Sanctions, debarment and entity identity

• Search OFAC sanctions lists
• Monitor OFAC sanctions list changes
• Build an OFAC vendor review workflow
• Scrape the UK sanctions list
• Scrape the UN sanctions list
• Compare UN sanctions regimes
• Build a vendor debarment watchlist
• Search cross-debarred suppliers
• Verify a company LEI for KYB
• Enrich counterparty data with LEI

Vulnerabilities and product safety

• Scrape the CISA KEV catalog
• Prioritise KEV remediation deadlines
• Scrape NVD CVE vulnerability data
• Monitor critical CVEs by product
• Scrape FDA food, drug and device recalls
• Monitor Class I FDA recalls
• Scrape CPSC product recalls
• Build a retail recall compliance feed

Public procurement

• Scrape India government tenders for bid tracking
• Monitor CPPP tender deadlines for compliance
• Find India government contracts by department

Browse everything at the blog hub.

Common patterns across compliance scrapers

Identifiers are the whole game. Every one of these sources has a canonical key: CIN for MCA, GSTIN for GST, CIK for SEC EDGAR, LEI for GLEIF, CVE ID for the NVD, recall number for FDA and CPSC. Store the identifier, never only the name. Name-based joins across registries produce false matches at a rate that makes downstream risk scores meaningless.

Validate before you look up. GSTIN, LEI, CIN and CVE IDs all have checkable structure. Rejecting malformed inputs locally turns a class of failed lookups into instant validation errors and cuts the volume you actually run.

Alias fields are not optional. Sanctions and debarment lists carry aliases, transliterations and former names because that is how designated parties appear in real documents. A screening implementation that matches only the primary name field will miss designations the list explicitly published.

Snapshot, hash and date every list. For any screening obligation, the defensible artefact is not "we checked" but "we checked against this exact version on this date". Persist the raw payload, a content hash and the retrieval timestamp for every list refresh. This is also what makes change detection trivial — diff two snapshots rather than re-reading the whole list.

Absence of a hit is a finding, and it needs provenance too. Record the negative result with the same source URL and timestamp you would record a hit with. Most audit friction comes from teams that log only positive matches.

Status fields drift silently. A company that was Active at onboarding may be Struck Off, Dormant or under liquidation a quarter later, and nothing notifies you. Re-run status checks on your existing vendor book on a schedule rather than only at onboarding.

Operational best practices for production pipelines

A handful of patterns matter more than the per-source specifics once you're running these scrapers in production at scale.

Tier the cadence to match signal half-life. Sanctions and KEV feeds justify daily refresh because designations and exploitation deadlines are time-bound. Registry status checks on an existing vendor book are quarterly work. Filings monitoring sits in between. Tier the watchlist into Tier 1 (screening obligations, daily), Tier 2 (active vendor book, weekly or monthly), Tier 3 (research, on demand).

Snapshot raw payloads alongside derived fields. Pipeline cost is dominated by scrape volume, not storage. Persisting the raw JSON response per snapshot lets you re-derive fields without re-scraping when your matching logic improves or your risk model changes — and in this category it doubles as your audit trail. Compress with gzip at write-time (4-8x size reduction).

Three-tier retention. Most production pipelines run: 90 days of raw snapshots at full fidelity + 12 months of derived per-entity records + indefinite retention of screening decisions and the list version each was made against. In regulated contexts the third tier is usually the one with a mandated minimum.

Cross-source dedup via identifiers, not names. Within a source, use its natural key. Across sources, resolve through a shared identifier — LEI where available, otherwise a registry number — and treat name similarity as evidence for review rather than as a join condition.

Validate live before declaring fields stable. Schemas drift, and official sources reformat without notice. Run a daily validation suite that asserts each source returns the expected core fields with non-null rates above 80% (for required fields) and 50% (for optional). Alert on schema breakage same-day.

Tag scrape timestamps in every record. Publication dates on official sources lag the underlying event, sometimes by weeks. Treat (published_date, retrieved_at) as a tuple — the first is when the authority acted, the second is when you could have known.