Track Talent Market Trends with LinkedIn Candidate Data

Thirdwatch's LinkedIn Candidate Finder lets you map talent supply by role, skill, and location across LinkedIn's 1B+ member profiles. Run systematic queries to measure how many qualified candidates exist per market, track changes over time, and identify emerging skill clusters. No LinkedIn login required. Built for HR analytics teams, workforce planners, and talent strategy leaders who need data-driven labor market intelligence.

Why track talent market trends with LinkedIn data

Workforce planning decisions -- where to open an office, which roles to fill locally versus remotely, whether a skill is scarce enough to justify a premium -- depend on understanding talent supply and demand. According to Korn Ferry's Future of Work research, the global talent shortage could reach 85 million workers by 2030, costing companies $8.5 trillion in unrealized annual revenue. The shortage is not uniform: some skills in some geographies are abundant while others face acute scarcity. The difference between guessing and knowing is data.

LinkedIn is the largest structured dataset of professional talent on the open web. Every profile is a data point: role, skills, location, seniority, employer. The challenge is accessing this data at scale without a LinkedIn Recruiter seat or Talent Insights subscription. The LinkedIn Candidate Finder turns structured queries into candidate counts and headline distributions -- the raw material for talent supply analysis. Run the same query across 10 cities and you have a supply heat map. Run it monthly and you have a trend line.

How does this compare to the alternatives?

Three approaches to talent market intelligence:

LinkedIn Talent Insights is the gold standard but requires enterprise pricing and a LinkedIn sales relationship. Government statistics (BLS in the US, ONS in the UK) are authoritative but lag by months and use broad occupational categories that do not map to specific technical skills like "Kubernetes" or "dbt". The Candidate Finder gives you real-time, skill-level granularity at per-result pricing -- the right tool for custom analyses that enterprise products do not template for you.

How to track talent trends in 4 steps

Step 1: How do I set up my API access?

Create a free account at apify.com, go to Settings, then Integrations, and export your API token.

export APIFY_TOKEN="apify_api_xxxxxxxxxxxxxxxx"

Step 2: How do I measure talent supply across locations?

Run the same role + skills query across multiple cities. The number of results returned approximates the publicly visible talent supply for that skill-location pair.

import os, requests, pandas as pd
from datetime import date

ACTOR = "thirdwatch~linkedin-candidate-finder-scraper"
TOKEN = os.environ["APIFY_TOKEN"]

CITIES = [
    "San Francisco", "New York", "Austin", "Seattle",
    "London", "Berlin", "Bangalore", "Toronto",
]

supply_data = []
for city in CITIES:
    resp = requests.post(
        f"https://api.apify.com/v2/acts/{ACTOR}/run-sync-get-dataset-items",
        params={"token": TOKEN},
        json={
            "role": "Machine Learning Engineer",
            "skills": ["pytorch", "python"],
            "location": city,
            "maxResults": 100,
        },
        timeout=300,
    )
    candidates = resp.json()
    supply_data.append({
        "city": city,
        "role": "ML Engineer",
        "skills": "pytorch, python",
        "candidate_count": len(candidates),
        "snapshot_date": str(date.today()),
    })

supply_df = pd.DataFrame(supply_data)
print(supply_df.sort_values("candidate_count", ascending=False))

The candidate_count is a proxy for publicly visible talent density, not an absolute measure. Cities with more LinkedIn-active professionals (San Francisco, London) will show higher counts. The value is in relative comparison across cities and over time.

Step 3: How do I compare skill density across roles?

Map which skills are abundant versus scarce in a specific market by running parallel queries across a skill matrix.

SKILLS_TO_TRACK = {
    "Kubernetes": ["kubernetes", "k8s"],
    "Machine Learning": ["machine learning", "pytorch"],
    "Data Engineering": ["spark", "airflow"],
    "Frontend React": ["react", "typescript"],
    "DevOps/SRE": ["terraform", "aws", "sre"],
}

TARGET_CITY = "Bangalore"

skill_density = []
for skill_label, skill_list in SKILLS_TO_TRACK.items():
    resp = requests.post(
        f"https://api.apify.com/v2/acts/{ACTOR}/run-sync-get-dataset-items",
        params={"token": TOKEN},
        json={
            "skills": skill_list,
            "location": TARGET_CITY,
            "country": "India",
            "maxResults": 100,
        },
        timeout=300,
    )
    candidates = resp.json()
    skill_density.append({
        "skill_cluster": skill_label,
        "location": TARGET_CITY,
        "candidate_count": len(candidates),
        "snapshot_date": str(date.today()),
    })

density_df = pd.DataFrame(skill_density)
print(density_df.sort_values("candidate_count", ascending=False))

Note that no role is specified here -- the query searches purely by skill, which gives a broader supply picture. Skills with high counts are abundant; skills with low counts may require premium compensation or remote hiring.

Step 4: How do I track trends over time with scheduled runs?

Schedule monthly snapshots for each skill-city pair. Over 3-6 months, the data reveals which markets are gaining or losing specific talent.

curl -X POST "https://api.apify.com/v2/schedules?token=$APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "talent-supply-ml-sf-monthly",
    "cronExpression": "0 9 1 * *",
    "timezone": "America/Los_Angeles",
    "isEnabled": true,
    "actions": [{
      "type": "RUN_ACTOR",
      "actorId": "thirdwatch~linkedin-candidate-finder-scraper",
      "runInput": {
        "role": "Machine Learning Engineer",
        "skills": ["pytorch", "python"],
        "location": "San Francisco",
        "maxResults": 200
      }
    }]
  }'

Store results in a time-series database or append to a CSV with a snapshot_date column. Plot candidate counts over time to detect supply trends -- rising counts in Austin may indicate tech migration from the Bay Area, while declining counts in a European city could signal visa policy impacts.

Step 5: How do I cross-reference supply with demand?

Pair candidate supply data with job posting data from the LinkedIn Jobs Scraper to build a supply-demand ratio.

# Supply: candidates with the right skills
supply_count = len(candidates)  # from the Candidate Finder run above

# Demand: open roles requiring those skills (from LinkedIn Jobs Scraper)
JOBS_ACTOR = "thirdwatch~linkedin-jobs-scraper"
jobs_resp = requests.post(
    f"https://api.apify.com/v2/acts/{JOBS_ACTOR}/run-sync-get-dataset-items",
    params={"token": TOKEN},
    json={
        "queries": ["machine learning engineer pytorch"],
        "location": "San Francisco",
        "maxResults": 100,
    },
    timeout=600,
)
demand_count = len(jobs_resp.json())

ratio = supply_count / max(demand_count, 1)
print(f"Supply: {supply_count} candidates, Demand: {demand_count} roles")
print(f"Supply/Demand ratio: {ratio:.1f}x")

A ratio below 2.0 signals a tight market -- expect longer time-to-fill, higher compensation requirements, and stronger competition for candidates. A ratio above 5.0 indicates an employer-favorable market where sourcing volume is not the bottleneck.

Sample output

A talent supply snapshot for ML Engineers across four cities produces data like this. The actor returns individual candidate profiles; you aggregate to get counts.

[
  {
    "fullName": "Sarah Kim",
    "headline": "ML Engineer at OpenAI | PyTorch, Distributed Training",
    "url": "https://www.linkedin.com/in/sarah-kim-ml/"
  },
  {
    "fullName": "Arjun Patel",
    "headline": "Senior Machine Learning Engineer at Flipkart | NLP, Python",
    "url": "https://www.linkedin.com/in/arjun-patel-ml-flipkart/"
  }
]

For trend analysis, the individual profiles matter less than the aggregate counts. But the headline field reveals qualitative signals: which companies employ the talent, what seniority level dominates, and which adjacent skills appear. A city where most ML Engineer headlines mention "NLP" but few mention "computer vision" tells you something about that market's specialization.

Common pitfalls

Three issues affect talent market analysis at scale. Confusing supply proxy with absolute count -- the candidate count from a maxResults=200 query is a proxy for relative density, not a census. Two cities both returning 200 results does not mean they have identical talent pools -- it means both hit your cap. Raise maxResults or use follow-up queries with narrower skill combinations to differentiate. Location variant leakage -- the actor expands common variants (Bangalore/Bengaluru, NYC/New York), but unusual city names or abbreviations may not match. When comparing cities, verify that your location strings are resolving correctly by spot-checking a few results. Snapshot timing bias -- LinkedIn profiles are indexed by public web search engines on varying schedules. A Monday morning snapshot and a Friday evening snapshot of the same query may return slightly different counts. Standardize your snapshot timing (first Monday of the month at 9 AM) for cleaner trend data.

The actor handles query construction, location variant expansion, and structured output so you can focus on the analysis layer rather than data collection mechanics.

Related use cases

• Scrape LinkedIn candidates for recruiter sourcing
• Build a talent pipeline from LinkedIn candidate data
• Find passive candidates on LinkedIn by skill
• Scrape LinkedIn jobs without login
• Build LinkedIn thought leadership dataset
• Track US tech hiring with Indeed data
• The complete guide to scraping jobs data
• All Thirdwatch use-case guides