Youtube transcript

YouTube Transcript (via DeepAPI, yt-dlp fallback)

Fetch a YouTube video's transcript and save a clean raw .txt file. Primary path is DeepAPI POST /v1/scrape/youtube/transcript — we dogfood David's own product. It runs server-side, so it avoids the local-IP bot flagging that plagues yt-dlp.

Save location

• If the user is in a real project/working dir → save there.
• Otherwise (no dir given, or cwd makes no sense) → save to ~/Downloads.
• Always name the file Channel_Title with spaces replaced by _ (e.g. David_Ondrej_title_of_video.txt). If metadata is unavailable, fall back to the video ID.

Primary path — DeepAPI

Key lives in ~/.zshrc as DEEPAPI_API_KEY. Do NOT source ~/.zshrc (breaks the shell, exit 126):

KEY=${DEEPAPI_API_KEY:-$(rg -o 'DEEPAPI_API_KEY=\S+' ~/.zshrc | head -1 | cut -d= -f2)}
BASE=${DEEPAPI_API_BASE_URL:-https://deepapi.co}

Run the scrape (keep the Idempotency-Key; retries must reuse the SAME one):

IDK=$(uuidgen)
curl -s --max-time 120 "$BASE/v1/scrape/youtube/transcript" \
  -H "Authorization: Bearer $KEY" \
  -H "Content-Type: application/json" \
  -H "Idempotency-Key: $IDK" \
  -d '{"url": "VIDEO_URL", "maxCostUsd": "0.05", "waitForFinishSecs": 60}' \
  > /tmp/yt_transcript.json

• Non-English videos: add "language": "de" (etc.) to the body.
• status: running → wait next.afterSecs, then curl "$BASE$(jq -r '.next.path' /tmp/yt_transcript.json)" -H "Authorization: Bearer $KEY" until succeeded or failed.

Extract the text and save it:

jq -r '.status' /tmp/yt_transcript.json                # succeeded | running | failed
jq -r '.output[0].text' /tmp/yt_transcript.json > "$OUT/$NAME.txt"
jq -r '.debitMicrousd' /tmp/yt_transcript.json         # cost (50000 = $0.05)

.output[0].segments also has timed segments (startSecs, durationSecs, text) if the user wants timestamps. Empty output = video has no captions; report it, don't retry.

For the Channel_Title filename, get metadata with a quick yt-dlp --print "%(channel)s|%(title)s" --skip-download "URL"; if that fails, use the video ID.

When to fall back to yt-dlp

• DEEPAPI_API_KEY missing from ~/.zshrc.
• HTTP 402 insufficient_credits (tell David to top up at deepapi.co/credits first; fall back only if he's unavailable).
• DeepAPI request failed twice.

Tell David whenever you fall back — a fallback means his product missed a real use case.

Fallback path — yt-dlp (local)

OUT="$(pwd)"            # or ~/Downloads if cwd makes no sense
META=$(yt-dlp --print "%(channel)s|%(title)s" --skip-download "URL")
NAME=$(echo "$META" | tr '| ' '__' | tr -cd '[:alnum:]_.-')   # "Channel_Title", spaces -> _, strip unsafe chars
yt-dlp --skip-download --write-subs --write-auto-subs \
  --sub-langs "en.*" --sub-format json3 \
  -o "$OUT/$NAME.%(ext)s" "URL"

• Fall back channel → uploader → uploader_id if channel is null.
• --skip-download = captions only. --write-subs + --write-auto-subs = manual first, auto as fallback.
• Always use json3, never VTT/SRT — auto VTT repeats every line twice (rolling captions).

Flatten json3 → raw text:

python3 - "$OUT" <<'PY'
import json, html, re, glob, sys, pathlib
f = glob.glob(sys.argv[1] + "/*.json3")
if not f: sys.exit("no json3 file")
data = json.load(open(f[0], encoding="utf-8"))
parts = ["".join(s.get("utf8","") for s in e.get("segs") or []) for e in data.get("events", [])]
txt = re.sub(r"\s+", " ", html.unescape(" ".join(p.strip() for p in parts if p.strip()))).strip()
out = pathlib.Path(f[0]).with_suffix(".txt")
out.write_text(txt, encoding="utf-8"); print(out)
PY

yt-dlp failure handling

• Non-English / unknown language: run yt-dlp --list-subs "URL" first, then set --sub-langs.
• Newer yt-dlp may need deno on PATH for YouTube extraction.
• On first failure: run yt-dlp -U once, retry once, then stop.
• 429 / "Sign in to confirm you're not a bot" = IP flagged. STOP — do NOT retry in a loop (makes it worse).
• Never fall back to downloading audio for Whisper unless the user explicitly asks.

Output

Report the saved path; print the text if short. If DeepAPI was used, also report the cost in dollars.