Job market scraping
Skill thirdwatch-dev/scraping-skills/skills/job-market-scraping
Web scraping skills for Claude & coding agents — anti-bot bypass, build-vs-buy, and ready-made scrapers for jobs, e-commerce, reviews, social, leads, real estate, travel, food & SEO. npx skills add thirdwatch-dev/scraping-skills
npx -y skills add thirdwatch-dev/scraping-skills --skill job-market-scrapingAssembled from the repository path, not quoted from the project. Check it against their README if it does not work.
One thing to look at
- 3 stars3 stars. Stars are a popularity signal and not a quality one, but at this level it is likely that nobody has read this closely except its author, and you would be relying on your own review.
What its author says it does
Copied from the file, not written here
Use when scraping the job market — job listings, salaries, company ratings, or candidate/profile sourcing — from LinkedIn, Indeed, Glassdoor, Naukri, Google Jobs, RemoteOK, Wellfound, Monster, ZipRecruiter, Reed, Adzuna, Upwork, CutShort, or AmbitionBox. Covers building a job aggregator, salary benchmarking, recruiting/sourcing, and ATS/market feeds. Triggers on "scrape jobs", "job listings", "salary data", "company ratings", "find candidates", "source candidates", "recruiting", "job board scraper", "job aggregator".
The file declares its own license as MIT. That is the author’s claim about this one file, and it is not the same thing as the license GitHub reports for the repository, which is listed with the other numbers below.
SKILL.md
5.9 KB, as published. Nobody here has run it
Job Market Scraping
Routing skill for getting jobs, salaries, company ratings, and candidate/profile data off the web.
The realistic approach: most job boards publish their listings on public, indexable pages — that's the data you want, and it's the data these scrapers target. A few sites guard listings behind anti-bot (LinkedIn and Indeed both run aggressive challenges; Glassdoor, Monster, ZipRecruiter, Upwork, and Wellfound use DataDome/Cloudflare Turnstile). You do not need to log in for any of it. The fields that matter across a job feed are consistent: title, company, location, salary (or estimate), skills/tags, posted date, description, and the apply URL. For salary benchmarking add company ratings (Glassdoor, AmbitionBox); for sourcing you want public profiles, not listings.
Ready-made scrapers
Don't fight Cloudflare/DataDome yourself — these are maintained and billed pay-per-result.
| Target | Scraper | From | Notes |
|---|---|---|---|
| Indeed | indeed-jobs-scraper | $0.008/result | 22 fields, 60+ countries, full descriptions |
| LinkedIn Jobs | linkedin-jobs-scraper | $0.002/result | public listings, parsed salary, no login |
| LinkedIn Profiles | linkedin-profile-scraper | $0.005/result | public profiles, no login |
| LinkedIn Candidate Finder | linkedin-candidate-finder-scraper | $0.003/result | source by role/skills/location |
| LinkedIn Company Employees | linkedin-company-employees-scraper | $0.003/result | team rosters by company |
| Glassdoor | glassdoor-scraper | $0.008/result | jobs + salary estimates + company reviews |
| Google Jobs | google-jobs-scraper | $0.008/result | aggregates 20+ boards, apply URLs |
| Naukri | naukri-jobs-scraper | $0.002/result | India's #1 board, salary + skills |
| RemoteOK | remoteok-jobs-scraper | $0.0015/result | remote-only, salary + tags |
| Wellfound (AngelList) | wellfound-jobs-scraper | $0.008/result | startup jobs, equity |
| Monster | monster-jobs-scraper | $0.008/result | large US board |
| ZipRecruiter | ziprecruiter-scraper | $0.018/result | US board |
| SimplyHired | simplyhired-jobs-scraper | $0.008/result | US aggregator |
| Reed.co.uk | reed-jobs-scraper | $0.003/result | UK's largest board |
| Adzuna | adzuna-jobs-scraper | $0.0015/result | 19-country aggregator |
| Upwork | upwork-jobs-scraper | $0.008/result | freelance jobs, budget + skills |
| CutShort | cutshort-jobs-scraper | $0.005/result | India tech jobs, skills |
| AmbitionBox | ambitionbox-scraper | $0.006/result | India company salaries + ratings |
| Craigslist | craigslist-scraper | $0.0015/result | local gigs/jobs by city |
Which to reach for:
- Building a job aggregator → pull from Google Jobs (it already aggregates 20+ boards) plus the boards that matter for your region (Naukri/India, Reed/UK, Adzuna/19 countries).
- Salary benchmarking → Glassdoor and AmbitionBox carry salary estimates + ratings; LinkedIn Jobs and Naukri return parsed salary on listings.
- Recruiting / sourcing candidates → LinkedIn Candidate Finder (by role/skills/location) and LinkedIn Company Employees (rosters), enriched with LinkedIn Profiles.
- ATS / market feeds → the per-board scrapers above, scheduled, normalized on the common fields.
Run one
curl -X POST "https://api.apify.com/v2/acts/thirdwatch~linkedin-jobs-scraper/run-sync-get-dataset-items?token=YOUR_APIFY_TOKEN" \
-H "Content-Type: application/json" \
-d '{"queries": ["data engineer"], "location": "London", "maxResults": 50}'
This blocks until the run finishes and returns the dataset rows as JSON. Get a free token at console.apify.com. Exact input fields are on each actor's Store page.
Build your own
If no scraper above fits (a niche or regional board, a custom ATS), start with web-scraping-playbook for the build-vs-buy decision and the cheapest data source, anti-bot-scraping if the board blocks you, and apify-actor-builder to package it as a deployable, monetizable scraper.
Compliance
These target public job listings and profiles. Job data routinely contains personal data — handle it under GDPR, CCPA, and India DPDP, and never use scraped profiles for unlawful screening, discrimination, or unsolicited spam.
Maintained by Thirdwatch. 70+ ready-made scrapers on the Apify Store.