The Complete Guide to Scraping Real Estate Listings (2026)

Thirdwatch publishes 9 real-estate scrapers covering US rentals (Apartments.com and Realtor.com), Zillow, Rightmove in the UK, five Indian portals, and Funda in the Netherlands. This guide is the decision tree for picking the right one — rent comps, underwriting, CMA reports, agent lead lists, market indices, and property-app backends.

The real-estate scraping landscape

Property data is the most geographically fragmented category on the web. There is no global portal: the United States splits between Zillow, Apartments.com and Realtor.com, the UK runs on Rightmove, India spreads across five large portals with meaningfully different inventory, and the Netherlands is effectively one site. A pipeline that works in Austin needs a completely different source list in Bangalore.

The demand for this data is structural rather than cyclical. The U.S. Census Bureau's Housing Vacancy Survey has kept the national rental vacancy rate near 7% through recent quarters, which means several million units are being priced, re-priced and concession-adjusted at any moment. Underwriters, revenue managers and proptech teams all need that pricing surface refreshed faster than any quarterly report delivers it.

For a US underwriting team, the right answer is usually US Rentals plus Zillow. For a UK yield model, Rightmove alone. For India, two portals minimum — inventory overlap between any pair of Indian sites is far lower than most teams expect.

Compare Thirdwatch real-estate scrapers

Scraper	Coverage	Modes	Pricing	Best for
US Rentals	US, Apartments.com + Realtor.com	Rent	Pay per result	Rent comps, lease-up tracking
Zillow Suite	US, nationwide	For sale, for rent, sold	Pay per result	CMA reports, yield screening
Rightmove	UK	Buy, rent	Pay per result	UK yield and agent data
MagicBricks	India, national	Rent, buy	Pay per result	Broad India coverage
99acres	India, national	Rent, buy, PG, commercial	Pay per result	Commercial and PG inventory
NoBroker	India, six metros	Rent, buy, PG	Pay per result	Owner-direct listings
Square Yards	India, national	Buy, rent	Pay per result	New projects and developers
CommonFloor	India, metro depth	Rent, buy, PG	Pay per result	Bangalore and Pune depth
Craigslist	US, by city	Housing and rooms	Pay per result	Long-tail private landlords

A Funda scraper covers Netherlands for-sale and rental listings — address, asking price, floor area, rooms, energy label, broker and listing status — and is available on request rather than as a published Store listing.

Decision tree

"I'm underwriting a US multifamily acquisition." US Rentals is the primary source. It returns price_min and price_max across floor plans, square_feet_min and square_feet_max, specials for concessions, and days_on_market — the four fields an underwriter actually needs to build effective rent rather than asking rent. Add Zillow Suite in forRent mode to catch scattered-site single-family rentals that Apartments.com under-indexes.

"I'm producing CMA reports for agents." Zillow Suite in sold mode. Sale price, date_sold, price_per_sqft, living_area_sqft, zestimate and tax_assessed_value are enough to assemble a defensible comparable set. Filter to sales within the last 6 months and within roughly one kilometre of the subject.

"I'm screening rental yield at portfolio scale." Zillow Suite in forSale mode with includeDetails enabled gives you list price and rent_zestimate on the same record, so gross yield is a single division. For the UK, Rightmove buy and rent runs over the same postcode outcodes produce the same ratio from real asking prices.

"I'm building an India rental price index." MagicBricks plus 99acres as the national spine, then NoBroker where you specifically want owner-direct pricing without brokerage inflation. Run each city separately; the city and localities inputs are how these actors partition inventory. CommonFloor adds depth in Bangalore and Pune where its locality coverage is stronger than the national portals.

"I'm building lead lists for a proptech or brokerage sales team." Zillow Suite with includeDetails returns listing-agent details; Rightmove returns agent_name and agent_phone directly; US Rentals returns leasing phone numbers and broker_name. Deduplicate on phone before any outbound sequence, and see the business data pillar for contact hygiene patterns.

"I'm backing a property search app with listing data." Zillow Suite plus US Rentals for the US, Rightmove for the UK, and two Indian portals per target city. Store the raw payload per snapshot and derive your own schema on top — portal field sets drift, and re-deriving from stored payloads is far cheaper than re-scraping.

Cross-source recipe: US rent comps normalized to price per square foot

import os, re, requests, pandas as pd

TOKEN = os.environ["APIFY_TOKEN"]


def run(actor, payload, timeout=3600):
    r = requests.post(
        f"https://api.apify.com/v2/acts/{actor}/run-sync-get-dataset-items",
        params={"token": TOKEN},
        json=payload,
        timeout=timeout,
    )
    return r.json()


ZIPS = ["78704", "78758", "78745"]

rentals = run(
    "thirdwatch~us-rentals-scraper",
    {"queries": ZIPS, "source": "both", "maxResults": 200, "minBeds": 1},
)
zillow = run(
    "thirdwatch~zillow-suite-scraper",
    {"queries": ZIPS, "searchMode": "forRent", "maxResults": 200},
)

r = pd.DataFrame(rentals)
z = pd.DataFrame(zillow)

# Apartments.com rows carry a band across floor plans; Realtor.com rows
# carry a single unit, so coalesce the band fields onto the single fields.
r["rent"] = r[["price_min", "price_max"]].mean(axis=1).fillna(r["price"])
r["sqft"] = r[["square_feet_min", "square_feet_max"]].mean(axis=1).fillna(
    r.get("square_feet")
)
r["beds"] = r.get("beds", r.get("beds_min")).fillna(r.get("beds_min"))
r = r[["listing_id", "source", "address", "zip_code", "rent", "sqft", "beds"]]

z = z.rename(
    columns={
        "zpid": "listing_id",
        "price": "rent",
        "living_area_sqft": "sqft",
        "bedrooms": "beds",
    }
).assign(source="zillow")
z = z[["listing_id", "source", "address", "zip_code", "rent", "sqft", "beds"]]


def addr_key(a):
    if not isinstance(a, str):
        return None
    return re.sub(r"[^a-z0-9]", "", a.split(",")[0].lower())


comps = pd.concat([r, z], ignore_index=True)
comps["addr_key"] = comps.address.apply(addr_key)
comps = comps.dropna(subset=["rent", "sqft"])
comps = comps[comps.sqft > 200]
comps = comps.drop_duplicates(subset=["addr_key", "beds"], keep="first")
comps["rent_psf"] = comps.rent / comps.sqft

print(
    comps.groupby(["zip_code", "beds"])
    .rent_psf.agg(["count", "median"])
    .round(2)
)

The address-plus-bedroom dedupe matters: the same building routinely appears on Apartments.com and Zillow with different names, and counting it twice biases a submarket median by several percent.

All use-case guides for real-estate scrapers

US Rentals (Apartments.com and Realtor.com)

• Build a US rent comps dataset for underwriting
• Analyze a metro rental market with US listings data
• Track rent concessions and lease-up specials
• Track rental days on market as a demand signal
• Build property-management lead lists from Apartments.com

Zillow

• Pull Zillow sold comps for CMA valuation reports
• Monitor Zillow price drops and new listings daily
• Screen rental yield with the Zillow Rent Zestimate
• Build a Zillow API alternative for a property app
• Build real-estate agent lead lists from Zillow

Rightmove (UK)

• Scrape Rightmove UK property listings
• Track the UK housing market with Rightmove
• Build a UK rental yield database from Rightmove

India portals

• Scrape MagicBricks for India property research
• Build a rental market database from MagicBricks
• Track India real-estate prices with MagicBricks
• Scrape 99acres India real-estate listings
• Monitor 99acres rental trends by city
• Build an India property research pipeline with 99acres
• Scrape NoBroker rentals without broker fees
• Track NoBroker vs MagicBricks India rentals
• Find India real-estate comps with Square Yards
• Build an India property price database from Square Yards
• Scrape CommonFloor for Bangalore real estate
• Track CommonFloor rental yield trends

Netherlands and long tail

• Scrape Funda property listings
• Track Amsterdam property prices on Funda
• Analyze Funda energy labels
• Build a Craigslist housing price monitor

Browse everything at the blog hub.

Common patterns across real-estate scrapers

Canonical natural keys. Zillow uses zpid. US Rentals, Rightmove, MagicBricks, 99acres and CommonFloor each issue a listing_id — but only stable within that portal, so always store (source, listing_id) as the composite key. Square Yards is best keyed on its canonical listing url.

A price is not a number until you make it one. Indian portals return both a display string and a numeric field: use rent_inr, price_inr or deposit_inr rather than parsing "₹45,000 per month", and never try to parse lakh and crore suffixes yourself. Rightmove separates price_value from price_qualifier ("Offers over", "Guide price") — the qualifier changes what the number means. US Rentals returns price_min and price_max for multi-floor-plan buildings and a single price for single units.

Carpet area versus super built-up area. This is the single largest source of bad India analysis. CommonFloor returns carpet_area_sqft and super_area_sqft separately, and they can differ by 25 to 40 percent for the same flat. Pick one convention for your whole dataset and convert everything to it, or your price-per-square-foot series will be meaningless.

Coordinates beat addresses. Most of these actors return latitude and longitude. For comp selection, radius filtering on coordinates is dramatically more reliable than string matching on addresses, which vary in formatting between portals for the same building.

Listing age is a first-class signal, not metadata. days_on_market, added_date, list_date and available_from are what turn a listing snapshot into a demand series. Snapshot daily and record disappearance dates too — a listing that vanishes is usually a lease signed, which is the strongest absorption signal available from public data.

Regulatory identifiers. Indian listings frequently carry a RERA registration ID, which is the only reliable way to tie a listing to a legally registered project. Filter on its presence when you need a defensible new-project dataset.

Operational best practices for production pipelines

A handful of patterns matter more than the per-actor specifics once you're running these scrapers in production at scale.

Tier the cadence to match signal half-life. Daily polling is canonical for monitoring use cases (price drops, new listings, concession changes), but most teams over-poll long-tail watchlist items. Tier the watchlist into Tier 1 (high-stakes, hourly), Tier 2 (active monitoring, daily), Tier 3 (research-only, weekly). Typical 60-80% reduction in run volume with negligible signal loss.

Snapshot raw payloads alongside derived fields. Pipeline cost is dominated by scrape volume, not storage. Persisting the raw JSON response per snapshot lets you re-derive metrics without re-scraping when your comp model improves, your submarket definitions change, or you discover a previously-ignored field. Compress with gzip at write-time (4-8x size reduction).

Three-tier retention. Most production pipelines run: 90 days of raw snapshots at full fidelity + 12 months of derived per-listing aggregates + indefinite retention of derived metric time-series. Storage costs stay flat at scale; query patterns map cleanly to the right retention tier.

Cross-source dedup via the canonical 4-tuple. Across-source dedup (Zillow vs Apartments.com; MagicBricks vs 99acres vs NoBroker) typically uses (address-norm, coordinate-proximity, bedroom-count, area-band). Within-source dedup uses each portal's stable natural key. Both are essential — get either wrong and your medians move.

Validate live before declaring fields stable. Schemas drift. Run a daily validation suite that asserts each scraper returns the expected core fields with non-null rates above 80% (for required fields) and 50% (for optional). Alert on schema breakage same-day.

Tag scrape timestamps in every record. Portal-displayed dates lag actual events by hours to days. For accurate freshness analysis, treat (platform_timestamp, scraped_at) as a tuple — the earlier is your "listed since" anchor, the later is your "still live as of" anchor.