Build a Country Market-Sizing Dataset

A production feed needs more than a scheduled scrape. The World Bank Indicators Scraper handles collection from the official World Bank Indicators API; the rest of the pipeline should make ownership and state explicit.

Recommended data flow

Saved Apify Task -> raw dataset -> keyed upsert -> qualification -> owner queue -> alert

Store each observation at country:indicator:year granularity. Keep a raw long table, then build a wide analytical view. This prevents a late revision to one indicator from rewriting unrelated source data.

Keep raw and derived data separate. The raw table should mirror source fields and collection metadata. The derived table can add scores, owners, workflow status, and internal notes. When a scoring rule changes, rebuild the derived table rather than editing source history.

Failure handling

Promote a snapshot only when the Actor run succeeds and passes basic checks: nonzero rows when rows are expected, valid keys, parseable dates, and no unexpected schema loss. Retry network failures with a bounded policy. Do not create downstream deletion events from a failed collection.

Coverage and release lag vary by series and country. Missing is not zero. Preserve null rows when auditing coverage, read the indicator definition, and never combine series until units and methodology are compatible.

Track three operational metrics: successful run rate, time from source publication to owner delivery, and the percentage of alerts that owners accept. Those numbers expose different failures. Availability problems hurt the first, an infrequent schedule hurts the second, and weak qualification hurts the third.

Start with one destination and one owner group. Add CRM, warehouse, or webhook branches after the keyed dataset and alert rules have stayed stable through several runs.