Wgetx

wgetx — 社交平台采集

Fetch data from supported social media platforms. Scripts ship inside this skill directory as pure ESM .mjs files under scripts/ (copied by npx skills add). Most scrapers use Node built-ins only — no npm install. Write outputs to /workspace/data/<platform>/ in Cohub, or ./data/<platform>/ elsewhere.

Install this skill

npx skills add https://github.com/markbang/wgetx-skill \
  --skill "wgetx" \
  --agent codex \
  --yes \
  --copy

Scripts are pure ESM .mjs on Node built-ins for most platforms. No package.json / npm install required for default HTTP scrapers.

Optional: only browser-based flows need Playwright:

cd .agents/skills/wgetx
npm init -y && npm i playwright && npx playwright install chromium

Pre-flight

# Pure HTTP scrapers: nothing to install (Node 18+ / 20+ / 24+)
node --version

# Optional browser scrapers only:
#   npm init -y && npm i playwright && npx playwright install chromium

which ffmpeg || echo "ffmpeg missing — --download-media/--transcribe will fail"
which whisper || echo "whisper missing — --transcribe will fail"

Quick Reference

Platform	Script	No Login?	Playwright?
一键搜索	social_search.mjs <kw>	✅	No
微博热搜/建议	weibo/weibo.mjs hot/search	✅	No
微博搜索/用户/详情	weibo/weibo_browser.mjs	❌	Yes
知乎热榜/用户/建议	zhihu/zhihu.mjs hot/user/suggest	✅	No
知乎搜索/问答	zhihu/zhihu.mjs search/question/answer	❌	Yes (login)
小红书笔记	xiaohongshu/xhs_fetch.mjs <id>	✅	No
小红书推荐流	xiaohongshu/xhs_feed.mjs	✅	No
抖音视频	`douyin/video_scraper.mjs <id	url>`	✅
微信搜索	wechat/wechat.mjs search <kw>	✅	Yes
贴吧论坛/帖子	tieba/tieba.mjs forum/thread	✅	Yes
X.com 用户/推文	x/x.mjs user/tweet	✅	No
Reddit 看板/搜索/帖子/评论/用户	reddit/reddit.mjs sub/search/post/comments/user	✅	No
B站搜索/视频/下载/评论/用户投稿	bilibili/bili.mjs search/video/download/comments/user	✅	No
快手视频	kuaishou/kuaishou_video.mjs <id>	✅	No
快手搜索/用户/评论	kuaishou/kuaishou_search.mjs	✅	Yes
YouTube Shorts/视频	youtube/shorts_scraper.mjs <id|url>	✅	No
Instagram 主页/帖子/评论	instagram/instagram.js <user|--post|--comments>	✅	Partial (--comments)
网易云音乐	netease/netease.mjs	❌	No

---

1. Web — 微博

No Login (Pure HTTP)

node scripts/weibo/weibo.mjs hot           # hot search (50 items)
node scripts/weibo/weibo.mjs hot 20        # top 20
node scripts/weibo/weibo.mjs search "AI"   # search suggestions

Output: /workspace/data/weibo/hot_search.json, search_suggest_<kw>.json

With Login (Playwright)

node scripts/weibo/weibo_browser.mjs login                              # QR login
node scripts/weibo/weibo_browser.mjs search "iPhone" 3                  # search weibos
node scripts/weibo/weibo_browser.mjs user 1642591402 2                  # user profile
node scripts/weibo/weibo_browser.mjs post 4960180420747996              # post detail
node scripts/weibo/weibo_browser.mjs timeline 3                         # home timeline

Cookies saved to /workspace/data/weibo/cookies.json.

---

2. Zhihu — 知乎

No Login

node scripts/zhihu/zhihu.mjs hot              # hot list (50)
node scripts/zhihu/zhihu.mjs hot 20
node scripts/zhihu/zhihu.mjs suggest "AI"     # suggestions
node scripts/zhihu/zhihu.mjs user zhang-jia-wei   # user profile

Output: /workspace/data/zhihu/hot_list.json

With Login (Playwright for QR, then HTTP)

node scripts/zhihu/zhihu_login.mjs                          # QR login
node scripts/zhihu/zhihu_login.mjs --cookie "z_c0=..."      # or inject cookie
node scripts/zhihu/zhihu.mjs search "AI" 5                  # search
node scripts/zhihu/zhihu.mjs question 267782048 3           # question + answers
node scripts/zhihu/zhihu.mjs answer 123456789               # answer + comments
node scripts/zhihu/zhihu.mjs topic 19550429 3               # topic feed

Cookies saved to /workspace/data/zhihu/cookies.json.

---

3. Xiaohongshu — 小红书

No Login (Pure HTTP SSR)

node scripts/xiaohongshu/xhs_fetch.mjs <note_id>
node scripts/xiaohongshu/xhs_fetch.mjs https://www.xiaohongshu.com/explore/xxx
node scripts/xiaohongshu/xhs_feed.mjs          # feed (20 items)
node scripts/xiaohongshu/xhs_feed.mjs 10

Output: /workspace/data/xiaohongshu/<note_id>.json, rednote_feed.json

---

Bilibili notes (updated)

# 视频详情（默认下载 mp4）
node scripts/bilibili/bili.mjs video BV1xx411c7mD
node scripts/bilibili/bili.mjs video BV1xx411c7mD --no-download

# 下载：多码率候选重试（360/480/720 + AVC 优先）
node scripts/bilibili/bili.mjs download BV1xx411c7mD
node scripts/bilibili/bili.mjs download BV1xx411c7mD --out-dir=/workspace/data/bilibili/out

# 用户投稿列表（WBI 签名，免登录；可多页）
node scripts/bilibili/bili.mjs user 546195
node scripts/bilibili/bili.mjs user 546195 3

# 评论稳路径：view 真 aid + /x/v2/reply
node scripts/bilibili/bili.mjs comments BV1xx411c7mD 3 hot

• 下载失败会自动换 DASH 候选；单流 HTTP 最多重试 3 次。
• 投稿列表优先 /x/space/wbi/arc/search，失败回退裸 arc/search。
• 免登录约到 720P；1080P+ 通常要 Cookie。

4. Douyin — 抖音

Requires Playwright. Browser-based X-Bogus signing. Douyin collection includes top-level comments only; second-level replies are outside scope.

node scripts/douyin/video_scraper.mjs <aweme_id|share_url|share_text> [output.json] [options]

node scripts/douyin/video_scraper.mjs 7644075801310154010
node scripts/douyin/video_scraper.mjs 'https://v.douyin.com/xxxx/'
node scripts/douyin/video_scraper.mjs '复制打开抖音... https://v.douyin.com/xxxx/'

Options: --comments-limit=500, --download-media, --transcribe

Output: /workspace/data/douyin/video_<aweme_id>.json (includes top-level comments)

Dependencies: ffmpeg for --download-media, whisper for --transcribe.

---

5. WeChat — 微信

Requires Playwright. Searches via Sogou Weixin.

node scripts/wechat/wechat.mjs search "AI" 3          # search articles
node scripts/wechat/wechat.mjs account 人民日报        # search accounts
node scripts/wechat/wechat.mjs article "<mp.weixin.qq.com url>"  # article detail
node scripts/wechat/wechat.mjs gzh "<__biz url>" 2     # account articles

Output: /workspace/data/wechat/search_<kw>.json, article_<name>.json

---

6. Tieba — 贴吧

Requires Playwright. Gets BAIDUID cookie, intercepts page_pc API.

node scripts/tieba/tieba.mjs forum 李毅            # forum threads
node scripts/tieba/tieba.mjs forum 显卡 3          # 3 pages
node scripts/tieba/tieba.mjs thread 7938723812     # thread + replies
node scripts/tieba/tieba.mjs thread 7938723812 5 lz   # only OP posts

Output: /workspace/data/tieba/forum_<name>.json, thread_<tid>.json

---

7. X.com — Twitter

Pure HTTP, no login. SSR parsing.

node scripts/x/x.mjs user elonmusk             # user + timeline (20 tweets)
node scripts/x/x.mjs user elonmusk 50          # 50 tweets
node scripts/x/x.mjs tweet 1812258574049157405 # tweet detail
node scripts/x/x.mjs tweet https://x.com/jack/status/20

Output: /workspace/data/x/x_user_<name>.json, x_tweet_<id>.json

⚠️ Search not available — X does not SSR search pages.

---

8. Reddit

双后端、纯 Node.js，无需浏览器：

• Pullpush（默认）：归档数据，任何 IP 可用
• Reddit direct（--live）：实时数据，需要非数据中心 IP；可选 OAuth 提限

# 看板帖子
node scripts/reddit/reddit.mjs sub programming 25
node scripts/reddit/reddit.mjs sub China 50

# 搜索 / 用户帖子
node scripts/reddit/reddit.mjs search "python async" 25
node scripts/reddit/reddit.mjs user spez 50

# 帖子详情 + 评论
node scripts/reddit/reddit.mjs post abc123
node scripts/reddit/reddit.mjs post https://reddit.com/r/Python/comments/abc123/
node scripts/reddit/reddit.mjs comments abc123 100

# Reddit 直连实时数据
node scripts/reddit/reddit.mjs sub programming --live 25

可选 OAuth（配合 --live，提高请求限额）：

REDDIT_CLIENT_ID=xxx REDDIT_CLIENT_SECRET=yyy \
  node scripts/reddit/reddit.mjs sub python --live 100

输出：/workspace/data/reddit/

命令	输出文件
sub <name>	reddit_sub_<name>.json
search <query>	reddit_search_<query>.json
user <name>	reddit_user_<name>.json
post <id>	reddit_post_<id>.json
comments <id>	reddit_comments_<id>.json

限制：Pullpush 数据可能延迟数周至数月且评论不完整；Reddit direct 可能屏蔽数据中心或 VPN IP。

---

9. Bilibili — B站

Pure HTTP, no login. Extensive public API coverage.

# Search
node scripts/bilibili/bili.mjs search 猫咪 5

# Video detail + download mp4 + danmaku text by default
# (cover/video/danmaku under data/bilibili/<bvid>/)
node scripts/bilibili/bili.mjs video BV1xx411c7mD
node scripts/bilibili/bili.mjs video 'https://b23.tv/xxxxxx'
node scripts/bilibili/bili.mjs video BV1xx411c7mD --no-download   # metadata + danmaku, no mp4
node scripts/bilibili/bili.mjs video BV1xx411c7mD --no-danmaku    # skip danmaku body
node scripts/bilibili/bili.mjs danmaku BV1xx411c7mD 2000          # danmaku only
node scripts/bilibili/bili.mjs download BV1rz7569EWw              # same default download
# note: 弹幕 ≠ 评论；danmaku is on-screen timeline text, comments are reply floors

# Comments (hot / newest)
node scripts/bilibili/bili.mjs comments BV1Wb411v7WN 5 hot

注意：评论稳路径：先用 view API 取真实 aid，再优先 `/x/v2/reply`；本地 bv2av 对新 BV 可能算错，`reply/main` 无 cookie 易 -352。免登录通常只能拿到热门置顶级少量评论。


# Discovery
node scripts/bilibili/bili.mjs hot              # hot words
node scripts/bilibili/bili.mjs popular          # popular videos
node scripts/bilibili/bili.mjs rank             # all-platform leaderboard
node scripts/bilibili/bili.mjs rank 188         # tech category

# User info
node scripts/bilibili/bili.mjs user 546195

Leaderboard categories: 0=全站, 1=动画, 3=音乐, 4=游戏, 5=娱乐, 36=知识, 188=科技

Output: /workspace/data/bilibili/

---

10. Kuaishou — 快手

Video Detail (Pure HTTP — No Browser)

node scripts/kuaishou/kuaishou_video.mjs 3xruk6a5qw3n6xq
node scripts/kuaishou/kuaishou_video.mjs https://www.kuaishou.com/short-video/xxx

Returns: video metadata, author info, music info, stats. No Playwright needed.

Search / User / Comments (Playwright)

node scripts/kuaishou/kuaishou_feed.mjs                        # home feed
node scripts/kuaishou/kuaishou_search.mjs search 美食 3        # search
node scripts/kuaishou/kuaishou_search.mjs user 李子柒 2        # user search
node scripts/kuaishou/kuaishou_user.mjs 3x984ye63jkct29       # user profile
node scripts/kuaishou/kuaishou_comment.mjs 3xruk6a5qw3n6xq    # comments
node scripts/kuaishou/kuaishou_full.mjs 3xruk6a5qw3n6xq       # full collection

Output: /workspace/data/kuaishou/

---

11. YouTube Shorts / Videos

基于 yt-dlp，无需登录、无需浏览器。自动下载 yt-dlp 到 /tmp/yt-dlp。

node scripts/youtube/shorts_scraper.mjs <video_url|video_id> [output.json] [options]

node scripts/youtube/shorts_scraper.mjs dQw4w9WgXcQ
node scripts/youtube/shorts_scraper.mjs 'https://www.youtube.com/shorts/xxxx'
node scripts/youtube/shorts_scraper.mjs 'https://youtube.com/shorts/xxxx?feature=share' /workspace/data/youtube/video.json

Options: --comments-limit=500, --with-replies, --replies-limit=100, --sub-langs=all, --no-auto-subs, --download-media, --transcribe

Capabilities:

• 视频元数据：标题、简介、标签、分类、时长、缩略图、热度曲线
• 统计数据：播放量、点赞数、评论数
• 频道信息：名称、ID、认证状态、粉丝数
• 一级评论（按热门排序，可选回复）
• 手工字幕 + 自动生成字幕（多语言 VTT 解析）
• --download-media 下载视频 + 提取 16k mono WAV
• --transcribe 下载 + whisper 转写

⚠️ 海量评论视频建议设较小的 --comments-limit。

Output: /workspace/data/youtube/<video_id>.json

---

12. Instagram

三种模式，全部无需登录。

用户主页（纯 HTTP）

node scripts/instagram/instagram.mjs <username> [output.json]

node scripts/instagram/instagram.mjs nasa
node scripts/instagram/instagram.mjs natgeo /workspace/data/instagram/natgeo.json

获取：主页信息（用户名/全名/简介/链接/头像/认证/业务类型）、粉丝/关注数、帖子/视频总数、最近 12 条图片帖 + 12 条视频帖（图片URL/视频URL/标题/点赞数/评论数/时间戳/播放数/时长）

单帖元数据（纯 HTTP，crawler UA）

node scripts/instagram/instagram.mjs --post '<url>'

node scripts/instagram/instagram.mjs --post 'https://www.instagram.com/p/xxx/'

获取：作者、标题/描述、点赞数、评论总数、图片URL、日期。基于 OG 元标签。

单帖评论（Playwright）

node scripts/instagram/instagram.mjs --comments '<url>'

node scripts/instagram/instagram.mjs --comments 'https://www.instagram.com/p/xxx/'

获取：评论内容（用户名 + 文本 + 时间），约 15 条顶部评论。依赖 Playwright 渲染 SSR HTML。

依赖：npx playwright install chromium（仅 --comments 模式需要）。

配合 WARP 代理防限流

bash /mods/warp-proxy/warp.sh
node scripts/instagram/instagram.mjs nasa --proxy socks5://127.0.0.1:10800

注意事项

• IP 频繁请求会被标记 require_login，建议间隔 ≥ 30s 或用 WARP 换 IP
• 评论仅约 15 条顶部可见，翻页需登录
• 私密账号仅返回基础信息
• 单个帖子详情/评论翻页/搜索均需登录

Output: /workspace/data/instagram/<username>.json

---

13. 网易云音乐 — NetEase Cloud Music

基于 weapi 加密的纯 HTTP API，无需浏览器。使用前需要扫码登录获取 MUSIC_U cookie。

# 首次使用：扫码登录
node scripts/netease/netease.mjs login

# 手动设置 cookie
node scripts/netease/netease.mjs cookie <MUSIC_U值>

# 搜索
node scripts/netease/netease.mjs search "晴天"
node scripts/netease/netease.mjs search "周杰伦" artist
node scripts/netease/netease.mjs search "摇滚" playlist 2

# 歌曲详情 + 歌词 + 热评
node scripts/netease/netease.mjs song 186016
node scripts/netease/netease.mjs song https://music.163.com/song?id=186016

# 歌词
node scripts/netease/netease.mjs lyric 186016

# 艺人详情 + 热门歌曲
node scripts/netease/netease.mjs artist 6452

# 专辑详情 + 曲目
node scripts/netease/netease.mjs album 189701

# 歌单详情 + 全部曲目
node scripts/netease/netease.mjs playlist 3097032173

# 榜单
node scripts/netease/netease.mjs toplist              # 全部榜单
node scripts/netease/netease.mjs toplist 3778678      # 榜单详情

# 首页轮播图
node scripts/netease/netease.mjs banner

# 评论
node scripts/netease/netease.mjs comment song 186016

# 相似歌曲
node scripts/netease/netease.mjs similar 186016

# 歌曲播放地址
node scripts/netease/netease.mjs url 186016
node scripts/netease/netease.mjs url 186016 higher

# 进阶组合
node scripts/netease/netease.mjs full 186016          # 歌曲全量
node scripts/netease/netease.mjs artist-full 6452     # 艺人全量
node scripts/netease/netease.mjs playlist-deep 3097032173  # 歌单深采

可获取数据

• 搜索：歌曲/艺人/专辑/歌单/歌词/用户/MV
• 歌曲详情：元数据、歌词（含翻译）、热评
• 艺人：简介、热门50首
• 专辑：信息、全部曲目
• 歌单：信息、全部曲目、标签
• 榜单：全部榜单列表、单个榜单详情+曲目
• Banner：首页轮播图
• 相似歌曲
• 播放地址（需登录 cookie）

注意事项

• 未登录大部分接口返回 code=-462 验证，需先 login 扫码获取 cookie
• Cookie 保存在 data/netease/cookies.json
• 纯 Node.js 内置模块，BigInt 手动 RSA，无外部依赖
• 搜索类型: song artist album playlist lyric user mv
• 播放等级: standard higher exhigh lossless hires

Output: /workspace/data/netease/

---

Social Search — 一键社媒搜索

Unified cross-platform search with single command. Default 4 HTTP platforms concurrent.

# Keyword search (4 platforms)
node scripts/social_search.mjs "人工智能"
node scripts/social_search.mjs "AI" --limit=10

# Hot mode (no keyword, trending across all platforms)
node scripts/social_search.mjs --all --limit=5

# Specific platforms
node scripts/social_search.mjs "科技" --platforms=bili,zhihu
node scripts/social_search.mjs --platforms=x --x-user=elonmusk
node scripts/social_search.mjs --platforms=douyin --douyin-link="https://v.douyin.com/xxx/"

# With Playwright platforms
node scripts/social_search.mjs "AI" --playwright --platforms=wechat,kuaishou,tieba

Available platforms: bili, weibo, zhihu, xhs, x, douyin, reddit, wechat, kuaishou, tieba

Output: /workspace/data/social_search/<key>_<ts>.json

---

Output Convention

All scripts write to /workspace/data/<platform>/. Unified JSON format with type, id, title, url fields where applicable.

Error Handling

Platform APIs are unstable. Failures return the HTTP status + endpoint. Individual platform failures in social_search.mjs do not block other platforms.