Diagnose Missing Emails in a Scraped Local Lead List (2026)

Roughly one business in four in a scraped local lead list comes back without an email address, and that is a real outcome rather than a broken run. The Google Maps Lead Enrichment actor stamps every row with fetch_status, http_server_header, tier_used, pages_fetched and contact_pages so you can separate "we never read this site" from "we read it and there is nothing there." Only the first group is worth re-running. This guide turns those fields into a triage table with one defined action per bucket.

Why "the scraper missed some" is not a single problem

The complaint arrives as one sentence: the lead list has holes. Underneath it are at least four unrelated failure modes with completely different economics.

A business whose domain no longer resolves is dead forever — re-running costs money and returns the same dns_error. A business whose edge protection served an empty shell page is recoverable on the second attempt for a few cents. A business whose site loaded perfectly and simply publishes a phone number instead of an email is not a failure at all; it is a routing decision. And a business with no website on its Maps listing never entered the crawl.

Across 118 real small-business websites measured in the US, UK, India and Australia, the actor found at least one email for 71–75% of businesses, at least one contact channel of any kind for ~95%, and hit unreachable sites 5–11% of the time. The email gap is dominated by verticals, not bugs: US home services land near 55–60% because they deliberately push click-to-call, while wellness and design businesses abroad land near 85–90%.

A separate, quieter cause is obfuscated markup. Sites behind Cloudflare's email address obfuscation replace addresses with an encoded hex string; 7 of 8 such sites in our sample returned zero emails to naive extraction. The actor decodes these, which is worth about six percentage points of hit rate on its own — so if a competing tool reported a bigger hole than this one, that mechanism is usually why.

How does this compare to the alternatives?

The difference between approaches is not whether they can fetch a homepage. It is whether the output tells you why a row is empty. A DIY crawler that returns email: null for a dead domain and email: null for a working site with no published address has thrown away the only information that would let you decide what to re-run. Generic scraping APIs return the raw HTML and leave classification to you, which means you rebuild the taxonomy yourself and maintain it as sites change.

How to diagnose missing emails in 4 steps

Which rows actually failed, and which ones just have no email?

Start by separating reachability from yield. Pull the dataset and bucket every row on fetch_status and the presence of primary_email. This single cross-tab usually resolves the argument on the spot.

from collections import Counter
from apify_client import ApifyClient

client = ApifyClient("<YOUR_APIFY_TOKEN>")
items = list(client.dataset("<YOUR_DATASET_ID>").iterate_items())

buckets = Counter()
for row in items:
    status = row.get("fetch_status")
    if status == "ok":
        if row.get("primary_email"):
            buckets["ok_with_email"] += 1
        elif row.get("primary_phone") or row.get("social_profiles_count"):
            buckets["ok_contactable_no_email"] += 1
        else:
            buckets["ok_but_empty"] += 1
    else:
        buckets[status] += 1

for name, count in buckets.most_common():
    print(f"{name:28} {count:5}  {count / len(items):.1%}")

In a healthy run, ok_with_email plus ok_contactable_no_email should account for roughly 85–90% of rows. If ok_but_empty is large, your queries are pulling businesses with template sites. If the non-ok statuses dominate, keep reading.

What does each fetch_status value mean, and what should I do about it?

Every non-ok status maps to exactly one action. Pair it with http_server_header, which tells you whether an edge network or the origin server refused you.

ACTION = {
    "blocked_shell":  "RE-RUN with retryBlockedWithProxy=true",
    "http_5xx":       "RE-RUN later, transient origin error",
    "timeout":        "RE-RUN once, then drop",
    "connect_error":  "RE-RUN once, then drop",
    "http_4xx":       "INSPECT http_status: 403 re-run, 404/410 drop",
    "dns_error":      "DROP, domain does not resolve",
    "ssl_error":      "DROP, certificate permanently broken",
    "not_html":       "DROP, website points at a file not a page",
    "social_website":  "ROUTE, facebook_url is already populated",
    "no_website":     "ROUTE to phone outreach, business_phone only",
}

for row in items:
    status = row.get("fetch_status")
    if status != "ok":
        print(status, row.get("http_status"), row.get("http_server_header"),
              row.get("website"), "->", ACTION.get(status, "review"))

How do I re-run only the recoverable slice?

Feed the recoverable URLs back in through websites, which skips the Google Maps discovery step entirely so you do not pay to rediscover businesses you already have. Turn on retryBlockedWithProxy for this pass only — it lifts reachability by roughly six percentage points and costs extra bandwidth, which is exactly the trade you want on a small, pre-filtered slice.

RECOVERABLE = {"blocked_shell", "http_5xx", "timeout", "connect_error"}

retry_urls = [
    r["website"] for r in items
    if r.get("fetch_status") in RECOVERABLE and r.get("website")
]
retry_urls += [
    r["website"] for r in items
    if r.get("fetch_status") == "http_4xx" and r.get("http_status") == 403
]

run = client.actor("thirdwatch/google-maps-lead-enrichment").call(run_input={
    "websites": retry_urls,
    "maxResults": len(retry_urls),
    "maxPagesPerSite": 4,
    "concurrency": 8,
    "verifyEmailDomains": True,
    "onlyWithContact": True,
    "retryBlockedWithProxy": True,
})

onlyWithContact: True on the retry pass drops businesses that still yield nothing, so the second run bills only for rows that improved. Check tier_used on the results: http means the plain fetch worked this time, http+tls or http+proxy means the retry path is what rescued the row.

Why does raising crawl depth past four pages not help?

Because the contact information is not deeper — it does not exist. maxPagesPerSite is capped at 4 on purpose. We tested ten pages per site against the businesses that produced no email at four: it recovered 2 of 18, and one of those was a regulator's registration address rather than the business's own inbox, while tripling bandwidth per site. Before you argue with the cap, confirm the budget was actually spent.

starved = [
    r for r in items
    if r.get("fetch_status") == "ok" and not r.get("primary_email")
]

for r in starved[:20]:
    print(r["business_name"],
          "pages:", r.get("pages_fetched"),
          "contact_pages:", len(r.get("contact_pages") or []),
          "bytes:", r.get("bytes_fetched"))

If pages_fetched is 4 and contact_pages already lists a /contact or /impressum URL, the crawler reached the page a human would click and found no address. That is a JavaScript contact form, not a depth problem. If pages_fetched is 1 with an empty contact_pages, the homepage exposed no internal links worth following — usually a single-page site, and again not depth-solvable.

How do I route phone-and-social-only rows instead of discarding them?

Split the dataset into outreach channels rather than filtering to email and throwing the rest away. Roughly 95% of businesses expose at least one channel, so a hard email filter discards a fifth of a list you already paid for.

email_ready, call_ready, social_ready, dead = [], [], [], []

for r in items:
    if r.get("primary_email") and r.get("email_domain_mx_ok"):
        email_ready.append(r)
    elif r.get("primary_phone"):
        call_ready.append(r)
    elif r.get("linkedin_url") or r.get("instagram_url") or r.get("facebook_url"):
        social_ready.append(r)
    else:
        dead.append(r)

print(len(email_ready), len(call_ready), len(social_ready), len(dead))

Note the email_domain_mx_ok guard on the first branch: about 5% of discovered email domains do not resolve at all because they are placeholders shipped with a website theme. Those belong in call_ready, not in your sending tool.

Sample output

Three redacted rows, one per diagnostic bucket. The business fields come from the Maps listing; the contact and diagnostic fields come from reading the website.

[
  {
    "business_name": "Austin Plumbing",
    "website": "https://austinplumbing.com/",
    "final_url": "https://www.austinplumbing.com/",
    "primary_email": "service@austinplumbing.com",
    "primary_email_source": "cfemail",
    "email_domain": "austinplumbing.com",
    "email_is_freemail": false,
    "email_domain_mx_ok": true,
    "email_mx_provider": "microsoft365",
    "primary_phone": "+15129004663",
    "social_profiles_count": 7,
    "contact_pages": ["https://www.austinplumbing.com/contact-us/"],
    "fetch_status": "ok",
    "http_status": 200,
    "http_server_header": "cloudflare",
    "tier_used": "http",
    "pages_fetched": 4,
    "has_contact": true
  },
  {
    "business_name": "Northside HVAC Co",
    "website": "http://northsidehvac.example/",
    "primary_email": null,
    "primary_phone": "+15125550142",
    "instagram_url": "https://www.instagram.com/northsidehvac/",
    "social_profiles_count": 2,
    "contact_pages": ["http://northsidehvac.example/contact/"],
    "fetch_status": "ok",
    "http_status": 200,
    "http_server_header": "LiteSpeed",
    "tier_used": "http",
    "pages_fetched": 4,
    "bytes_fetched": 288104,
    "has_contact": true
  },
  {
    "business_name": "Riverbend Dental",
    "website": "https://riverbenddental.example/",
    "primary_email": null,
    "primary_phone": "+15125550188",
    "fetch_status": "blocked_shell",
    "http_status": 200,
    "http_server_header": "cloudflare",
    "tier_used": "http",
    "pages_fetched": 1,
    "bytes_fetched": 1483,
    "has_contact": true
  }
]

Row one is a clean hit, and primary_email_source: "cfemail" records that the address came out of obfuscated markup — a row a simpler tool would have reported as empty. Row two is the honest miss: four pages read, a real /contact/ page among them, no address published, phone and Instagram still usable. Row three is the recoverable case — bytes_fetched of 1,483 against an HTTP 200 is the shell-page signature, which is why status is decided on payload size rather than status code.

Common pitfalls

Treating every empty email as a retry candidate. Re-running dns_error, ssl_error and not_html rows bills you again for an outcome that cannot change. Filter to the recoverable set first.

Assuming MX validation means deliverability. email_domain_mx_ok is domain-level. It proves the domain accepts mail; it does not prove info@ exists. Mailbox confirmation needs an SMTP conversation this actor deliberately does not attempt.

Filtering out free-mail addresses automatically. About one in six small businesses uses a Gmail or Yahoo address as its genuine business contact. email_is_freemail is a flag, not a verdict.

Raising maxResults on one broad query. A single Maps query saturates near 120 businesses. Past that you are re-paging the same places. Split by suburb or sub-category instead — the city-by-city coverage guide covers the non-overlapping partition.

Benchmarking one vertical against another. A 58% email rate on US plumbers and an 88% rate on Melbourne yoga studios are both normal. Set expectations per vertical before you promise a number to anyone.

Each of these is visible in the output rather than hidden: fetch_status, http_server_header, tier_used, pages_fetched and email_domain_mx_ok exist specifically so a run can be audited after the fact instead of argued about.

Related use cases

• Build a city-by-city local lead list from Google Maps — non-overlapping coverage across suburbs and trades without paying twice.
• Cut cost per contactable lead in local prospecting — what a usable lead actually costs once the empty rows are priced in.
• Segment leads by email provider: Workspace vs Microsoft 365 — turn email_mx_provider into cohorts with different angles.
• Measure local business digital presence by city and vertical — the same reachability fields, used as a market statistic.
• Google Maps Lead Enrichment actor page and the business data scraping guide.
• More engineering write-ups on the Thirdwatch blog.