Build an EU Procurement Lead Pipeline

A production feed needs more than a scheduled scrape. The EU TED Tenders Scraper handles collection from the European Union's official TED Search API v3; the rest of the pipeline should make ownership and state explicit.

Recommended data flow

Saved Apify Task -> raw dataset -> keyed upsert -> qualification -> owner queue -> alert

Use publication_number as the idempotency key. Store raw notices first, derive a lead table second, and route only open opportunities to CRM owners. A daily Task is normally enough because TED is a publication feed, not a live auction.

Keep raw and derived data separate. The raw table should mirror source fields and collection metadata. The derived table can add scores, owners, workflow status, and internal notes. When a scoring rule changes, rebuild the derived table rather than editing source history.

Failure handling

Promote a snapshot only when the Actor run succeeds and passes basic checks: nonzero rows when rows are expected, valid keys, parseable dates, and no unexpected schema loss. Retry network failures with a bounded policy. Do not create downstream deletion events from a failed collection.

TED text is multilingual. The Actor prefers English where the notice supplies it, but some titles remain in another source language. Buyers and CPV codes are arrays because one notice can contain several of each.

Track three operational metrics: successful run rate, time from source publication to owner delivery, and the percentage of alerts that owners accept. Those numbers expose different failures. Availability problems hurt the first, an infrequent schedule hurts the second, and weak qualification hurts the third.

Start with one destination and one owner group. Add CRM, warehouse, or webhook branches after the keyed dataset and alert rules have stayed stable through several runs.