Paywall bypass skill
Identify paywall types and generate bypass strategies for web scraping and content extraction.
npx -y skills add Lunatic16/paywall-bypass-skillAssembled from the repository path, not quoted from the project. Check it against their README if it does not work.
2 things to look at
- no licenseNo license file was found in the repository. Code published without one is not open source by default, so using it at work is a question for whoever answers licensing questions where you are.
- 5 stars5 stars. Stars are a popularity signal and not a quality one, but at this level it is likely that nobody has read this closely except its author, and you would be relying on your own review.
What its author says it does
Copied from the file, not written here
Identify paywall types and generate bypass strategies for web scraping and content extraction. Covers 500+ news sites, major paywall providers (Piano, TinyPass, Poool, etc.), and extraction techniques (bot spoofing, AMP, JSON-LD, archive fetching).
SKILL.md
8.7 KB, as published. Nobody here has run it
Paywall Bypass Helper
Identify paywall provider types and generate bypass strategies for web scraping, content extraction, and research across 500+ news sites.
Use this skill when
- The user asks to "bypass paywall", "extract article content", or "get full text" from a paywalled URL
- The user needs to scrape or research content from news sites with subscription walls
- The user asks what paywall technology a specific site uses
- The user needs to write scripts that handle paywalled content programmatically
- The user asks about cookie behavior that triggers or avoids paywalls
Do not use this skill when
- The task involves cracking DRM, decrypting encrypted content, or circumventing hard encryption
- The user requests illegal access to credentials or personal data
- The target is a non-paywall access control (login-only portals, intranets)
- The user wants to distribute full paywalled content commercially
How paywalls work
Paywalls generally work through one of these mechanisms:
- Metered paywalls — track article views via cookies/localStorage; after N free articles, block access
- Hard paywalls — content is loaded but hidden behind CSS/JS overlay; blocking the JS reveals text
- Server-side paywalls — content is not sent to the browser; requires external fetching (archive, cache, bot UA)
- Hybrid paywalls — combination of client-side blocking + server-side checks
Paywall provider identification
To identify which paywall provider a site uses, inspect the page source for these signatures:
| Provider | Detection Script/Pattern | Sites Using |
|---|---|---|
| Piano.io | .piano.io/, .piano.io/xbuilder/ | Foreign Policy, DN.no, Funke, Vocento, Berlingske |
| TinyPass (Piano) | .tinypass.com/, js.tinypass.com | Chicago Sun-Times, Fortune, Adweek, Corriere.it, Digiday |
| Poool.fr | .poool.fr/ | Alternatives Economiques, Challenges, Elle.fr, Le Télégramme, GCF group |
| Sophi.io | .sophi.io/ | Adweek, Automobilwoche, Business Insider, Crain Comm |
| Evolok | .evolok.net/ | El País, Vocento group, Leaky Paywall |
| Qiota | .qiota.com/ | AutoPlus.fr, Cosmopolitan.fr, Nice-Matin group, Rossel group |
| Pelcro | js.pelcro.com/ | Domani, Foreign Affairs, Crain Comm |
| Cxense | .cxense.com/ | Barron's, Business Insider JP, IPM group |
| Blueconic | .blueconic.net/ | Bridge Tower Media |
| Memberstack | [data-memberstack-app] | Various SaaS-gated sites |
| Steady | steadyhq.com/ | American Purpose, some Ghost sites |
| Leaky Paywall | /leaky-paywall/ | WordPress sites with Leaky plugin |
| Wallkit | wallkit.net | WordPress subscription sites |
| AMP Access | .ampproject.org/v0/amp-access-, .ampproject.org/v0/amp-subscriptions- | Many sites with AMP versions |
For a complete list of patterns, see references/paywall-patterns.md.
Bypass technique decision tree
Step 1: Identify the paywall type
Check page source for:
1. Scripts matching known paywall providers (table above)
2. Cookie names (TDNotesRead, TID_ID, ra, blaize_session, etc.)
3. AMP version availability (<link rel="amphtml">)
4. JSON-LD article data (script[type="application/ld+json"])
5. CSS class patterns (paywall, overlay, modal, gated)
Step 2: Choose bypass technique
| Technique | When to use | How |
|---|---|---|
| Block paywall script | Script-based paywall (TinyPass, Piano, Poool, etc.) | Block the script URL; content loads but paywall overlay doesn't trigger |
| Clear cookies | Metered paywall tracking views | Remove tracking cookies; localStorage.clear() + sessionStorage.clear() |
| Bot UA spoofing | Server serves full content to crawlers | Set User-Agent to Googlebot/Bingbot; add Referer: https://www.google.com/ |
| AMP page | Site has AMP version with relaxed paywall | Navigate to AMP URL or unhide content blocked by amp-access/amp-subscriptions |
| JSON-LD extraction | Article body in structured data | Parse script[type="application/ld+json"] for articleBody key |
| Archive fetch | Server-side paywall, no client-side text | Fetch article from archive.is or Google cache |
| Next.js data | Sites using Next.js SSR | Extract from script#__NEXT_DATA__ |
| External link | No bypass possible client-side | Generate link to archive.is or Google Search Tool for manual fetch |
Step 3: Implement bypass
For each technique, see references/content-extraction.md for implementation details including code patterns, selectors, and fallback strategies.
Cookie management per provider
Different paywall providers use different cookies to track access:
| Provider | Tracking Cookie(s) | Action |
|---|---|---|
| Piano (general) | None specific | Block script, no cookie action needed |
| TinyPass (Piano) | xbc (HBR) | Drop specific cookie |
| DPG Media (NL) | TID_ID | Drop this cookie to reset meter |
| Haaretz Group | ra | Drop this cookie |
| Adweek/Zephr | blaize_session | Drop this cookie |
| Pitchfork | pay_ent_msmp | Drop this cookie |
| Business Standard | userUid | Drop this cookie |
| eNotes | ENOTESID | Drop this cookie |
| DN Media (no) | AnonUserCookie | Drop this cookie |
Important: Some sites require cookies to be preserved (e.g., authentication-based bypass). Check references/cookie-rules.md for per-site rules.
Common media groups
Many news sites share paywall infrastructure. Identifying the media group simplifies bypass:
| Group | Key Sites | Paywall Type |
|---|---|---|
| News Corp Australia | theaustralian.com.au, heraldsun.com.au, dailytelegraph.com.au | AMP subscriptions + hard paywall |
| McClatchy (USA) | bnd.com, miamiherald.com, sacbee.com, kansascity.com | Piano + Googlebot |
| Gannett | azcentral.com, freep.com, indystar.com | Googlebot UA |
| Hearst (USA) | houstonchronicle.com, sfchronicle.com, statesman.com | Custom script blocking |
| Lee Enterprises | townnews.com-based sites | TownNews/Leaky paywall |
| Vocento (ES) | abc.es, elmundo.es (not), larioja.com, ideal.es | Evolok + AMP access |
| GEDI (IT) | repubblica.it, lastampa.it, huffingtonpost.it | Piano + Googlebot |
| DPG Media (NL) | volkskrant.nl, trouw.nl, demorgen.be | Custom temptation.js |
| Groupe Rossel (FR/BE) | lavoixdunord.fr, lesoir.be, sudinfo.be | Qiota |
| Funke (DE) | abendblatt.de, morgenpost.de, waz.de | Piano |
| Crain Comm (USA) | adage.com, autonews.com, chicagobusiness.com | Pelcro + Sophi |
| DN Media (NO) | dn.no, rechargenews.com, upstreamonline.com | Custom + fetch from archive |
| Conde Nast (USA) | newyorker.com, vogue.com, wired.com, gq.com | Custom + script blocking |
| Advance Local (USA) | nj.com, mlive.com, cleveland.com, al.com | Sophi + DOMPurify |
Implementation guidelines
When writing a scraper/bypass script
- Always inspect first — check page source for paywall scripts before attempting bypass
- Start with the least invasive technique — cookie clear > script block > UA spoof > external fetch
- Handle JavaScript-rendered content — many modern sites use React/Next.js; use Puppeteer/Playwright if needed
- Respect rate limits — do not hammer servers with requests; add delays between fetches
- Use headless browsers for JS-rendered paywalls — some paywalls only trigger after JS execution
- Validate output — verify extracted text is complete, not truncated teaser content
When analyzing a new/unlisted site
- Check for AMP version:
<link rel="amphtml" href="..."> - Search for paywall scripts:
.piano.,.tinypass.,.poool.,.sophi.,.evolok.,.qiota. - Check localStorage keys for meter tracking
- Try Googlebot UA:
Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) - Check for JSON-LD:
script[type="application/ld+json"]witharticleBody - Try archive.is as fallback
References
references/paywall-patterns.md— Complete regex patterns for all paywall providersreferences/content-extraction.md— JSON-LD, AMP, archive.is, Next.js extraction methodsreferences/cookie-rules.md— Per-site cookie management rules
Examples
examples/identify-paywall.md— Walkthrough for detecting paywall type from HTML sourceexamples/bypass-strategy.md— Step-by-step bypass recipe for a specific site