Open-Source Web Scraping Skills for Claude Code Agents

Thirdwatch open-sourced twelve web scraping skills for Claude Code and other AI coding agents. Three are pure engineering (a build-vs-buy playbook, an anti-bot bypass guide, and how to ship an Apify actor); nine route a request like "scrape Indeed jobs" or "get product prices from Flipkart" to a working approach. Install all twelve with npx skills add thirdwatch-dev/scraping-skills. MIT licensed, free, and usable in any agent that supports the skills format.

Why we built scraping skills

If you point an AI coding agent at "scrape this site," it usually does the naive thing: spin up a headless browser and start clicking. That is the slowest, most expensive, and most fragile option — and it still fails on any site with real bot defenses. The agent doesn't know the cheaper paths, doesn't know which sites need a browser at all, and doesn't know when a maintained scraper already exists.

That knowledge is exactly what a skill is for. A skill is a short markdown file an agent loads on demand when your request matches its description — domain expertise injected at the right moment, with no copy-paste.

The problem is big enough to be worth encoding. Cloudflare alone sits in front of around one in five of all websites, and that's before counting the other major anti-bot vendors. A scraper that works today routinely breaks next month when a protection changes. Most teams rediscover the same hard-won lessons one painful 403 at a time. We wrote those lessons down.

What's in the pack

Twelve skills, split into the how and the what.

Engineering skills — how to scrape anything:

• web-scraping-playbook — the decision framework. Find the cheapest data source first (an API or the page's embedded JSON beats parsing HTML), then climb a cost-first ladder only as far as you must. Includes the build-vs-buy call and compliance basics.
• anti-bot-scraping — what to do when a site blocks you. How to tell a transient network failure from a real block, then a cheapest-first bypass ladder from a plain HTTP request up to a full production browser, plus an honest list of sites that resist every affordable approach.
• apify-actor-builder — how to package a scraper as a deployable, schedulable, billable Apify actor, including the input/output schemas that trip most people up.

Domain task-routers — what to scrape:

Install in one line

The skills use the open npx skills tooling. From any project:

npx skills add thirdwatch-dev/scraping-skills

That pulls all twelve and wires them into Claude Code (and a long list of other agents) automatically. From then on, when you ask your agent to scrape something, the matching skill loads itself — you don't have to name it. The source is on GitHub under an MIT license, so you can read every line before you run it, or fork and trim it to your own stack.

The idea baked into every skill: don't fight the treadmill twice

The skills are genuinely useful on their own — but they all carry the same conclusion. Maintaining a scraper against an evolving bot defense is a treadmill. Doing it across dozens of sites is a full-time job. So where a maintained scraper already exists, the skill points to it.

Here's the honest trade-off the skills lay out:

That's why the domain skills route to Thirdwatch's catalogue of 70+ scrapers on the Apify Store. They run on transparent pay-per-result pricing, the anti-bot work and proxy rotation are handled, and you pay only for the rows you get back. If you'd rather build it yourself, the engineering skills show you how — both options are first-class.

From "scrape Indeed jobs" to data

A concrete flow. You tell your agent: "Get me 50 data-scientist jobs from Indeed in Berlin." The job-market-scraping skill loads, recognizes Indeed as a heavily protected target, and reaches for the maintained path instead of a brittle browser script:

curl -X POST "https://api.apify.com/v2/acts/thirdwatch~indeed-jobs-scraper/run-sync-get-dataset-items?token=YOUR_APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{ "queries": ["data scientist"], "location": "Berlin", "maxResults": 50 }'

You get clean JSON rows back in one call — titles, companies, salaries, descriptions — with none of the bot-defense fight. Each actor's exact input fields live on its Store page, linked from the skill. Prefer to roll your own? The same skill hands you off to anti-bot-scraping and apify-actor-builder.

What we kept to ourselves

In the spirit of being useful without being naive: the skills teach technique, which is largely public knowledge anyway. What they deliberately don't publish is our internal economics — the per-site cost math, the exact tuning that makes a given scraper cheap to run, and the operational machinery that keeps 70+ actors alive. The difficulty of the problem is on full display; the answer key to our unit economics is not. That's the right line for an open-source release: give away the map, keep the mine.

Related reading

• A practical guide to scraping e-commerce sites
• A practical guide to scraping business and local data
• Build a local business database with Google Maps
• Build a LinkedIn jobs aggregator with Apify
• Browse the full Thirdwatch blog