Video clipper

Video Clipper

Takes a long-form video and produces ready-to-post short-form vertical clips with speaker-tracked framing and professional animated captions. Works with podcasts, interviews, talks, and any talking-head content.

---

Requirements

• FFmpeg installed and available in PATH (brew install ffmpeg on macOS, apt install ffmpeg on Linux)
• Python 3 with openai-whisper and requests packages (pip install openai-whisper requests). Note: openai-whisper installs PyTorch (~2GB download). This skill uses openai-whisper instead of the lighter whisper-cpp because it provides word-level timestamps needed for accurate viral moment scoring.
• yt-dlp installed (for YouTube/URL downloads) — brew install yt-dlp on macOS, pip install yt-dlp on Linux
• API Keys in .env file (project root or any parent directory):
  • KLAP_API_KEY — from klap.app (reframing with speaker tracking)
  • CAPTIONS_AI_API_KEY — from captions.ai / platform.mirage.app (animated captions)

Before starting: Verify that FFmpeg, yt-dlp, and the Python packages are installed. If any are missing, instruct the user to install them before proceeding.

Cost Per Clip

Step	Cost
Whisper (transcription)	Free (local)
FFmpeg (clip extraction)	Free (local)
Klap (reframing)	~$1.50-2.50/clip depending on plan
Captions.ai (captions)	~$0.15/min of output
Total per clip	~$2-3

---

Input

The user provides:

1. Video source (required) — one of:

   • Local file path — e.g. /path/to/podcast.mp4
   • YouTube URL — e.g. https://www.youtube.com/watch?v=...
   • Any public video URL — direct link to MP4

2. Moment selection mode (ask the user):

   • Automatic — Claude picks the best moments
   • Manual — user provides specific timestamps
   • Hybrid — Claude proposes moments, user approves/adjusts before processing

3. Number of clips (optional) — default 3-5. Depends on video length and content density.

4. Caption template (optional) — Captions.ai template ID. Default: ctpl_DxflLOnuKkb198FNdI9E (Heat). List available templates via the API if user wants to browse.

5. Target clip duration (optional) — default 15-60 seconds. User can specify a range.

---

Pipeline

Step 1: Get the Video

Based on input type:

Local file:

# Verify it exists and get duration
ffprobe -v quiet -print_format json -show_format "video.mp4"

YouTube URL:

yt-dlp -f "bestvideo[height<=720]+bestaudio/best[height<=720]" --merge-output-format mp4 -o "<workdir>/source.mp4" "<URL>"

Other URL:

curl -L -o "<workdir>/source.mp4" "<URL>"

Step 2: Transcribe with Whisper

import whisper

model = whisper.load_model("base")
result = model.transcribe("source.mp4", language="en", word_timestamps=True)

Save both:

• transcript.json — full result with word-level timestamps (needed for Step 3)
• transcript.txt — readable version with timestamps per segment (for Claude to analyze)

Step 3: Identify Best Moments (Viral Scoring)

This is the key intelligence step. Claude reads the full transcript and identifies potential clip moments.

Step 3a: Segment the transcript into candidate moments

Scan the transcript for self-contained 15-60 second windows. Look for natural start/end points (topic changes, pauses, complete thoughts).

Step 3b: Score each candidate moment on this rubric

For each candidate, score 1-10 on these five criteria:

Criteria	What to look for	Score guide
Hook Strength	Does the first sentence grab attention? Is it a surprising claim, provocative question, or bold statement?	10 = "wait, what?" reaction. 1 = generic setup
Quotability	Contains a memorable one-liner that people would screenshot or share?	10 = tweet-worthy standalone quote. 1 = no standalone phrases
Emotional Intensity	Does the speaker show passion, humor, anger, vulnerability, or conviction?	10 = genuine emotion. 1 = monotone/flat delivery
Self-Containedness	Does it make complete sense without watching the rest of the video?	10 = fully standalone. 1 = needs prior context
Surprise/Controversy	Does it challenge conventional wisdom, reveal something unexpected, or take a hot take?	10 = counterintuitive insight. 1 = commonly known information

Total score = sum of all five (max 50).

Step 3c: Rank and select top N moments

• Sort by total score descending
• Select top N (user-specified or default 3-5)
• Ensure selected moments don't overlap
• Prefer variety in topics/angles — don't pick 3 clips about the same point

Step 3d: Present to user for approval

For each selected moment, show:

• Timestamp range (start - end)
• Duration
• Transcript excerpt (first 2-3 lines)
• Score breakdown (hook/quotability/emotion/self-contained/surprise)
• Total score
• Suggested hook text for the clip

Wait for user approval. User can:

• Approve all
• Remove specific clips
• Add their own timestamps
• Adjust start/end times
• Request more options

Do NOT proceed to Step 4 until user approves.

Step 4: Extract Raw Clips

For each approved moment, extract with FFmpeg:

ffmpeg -y -ss <start> -to <end> -i source.mp4 -c copy clip<N>-raw.mp4

Step 5: Reframe with Klap

Upload each raw clip to Klap for AI-powered speaker-tracked reframing to 9:16.

API: Klap

• Endpoint: POST https://api.klap.app/v2/tasks/video-to-video
• Auth: Authorization: Bearer <KLAP_API_KEY>

Submit each clip:

import requests

headers = {
    "Authorization": f"Bearer {klap_key}",
}

# Direct file upload
with open("clip-raw.mp4", "rb") as f:
    r = requests.post(
        "https://api.klap.app/v2/tasks/video-to-video",
        headers=headers,
        files={"video": f},
        data={
            "language": "en",
            "editing_options": '{"captions":false,"reframe":true,"emojis":false,"intro_title":false}',
            "dimensions": '{"width":1080,"height":1920}'
        }
    )
task_id = r.json()["id"]
output_id = r.json().get("output_id")

Poll until ready:

# Poll every 30 seconds
r = requests.get(f"https://api.klap.app/v2/tasks/{task_id}", headers=headers)
status = r.json()["status"]  # "processing" or "ready"
output_id = r.json()["output_id"]  # project ID when ready

Export the reframed video:

# Request export
r = requests.post(
    f"https://api.klap.app/v2/projects/{output_id}/exports",
    headers=headers,
    json={}
)
export_id = r.json()["id"]

# Poll export every 15 seconds
r = requests.get(
    f"https://api.klap.app/v2/projects/{output_id}/exports/{export_id}",
    headers=headers
)
# When status != "processing", download from src_url
download_url = r.json()["src_url"]

Klap handles:

• Face detection and tracking
• Active speaker detection (for multi-person videos)
• Smooth 16:9 → 9:16 reframing
• Dynamic cropping that follows the speaker

Step 6: Add Animated Captions with Captions.ai

Upload each reframed clip to Captions.ai for professional animated captions.

API: Captions.ai (Mirage)

• Endpoint: POST https://api.mirage.app/v1/videos/captions
• Auth: x-api-key: <CAPTIONS_AI_API_KEY>

Submit each clip:

headers = {"x-api-key": captions_key}

with open("clip-reframed.mp4", "rb") as f:
    r = requests.post(
        "https://api.mirage.app/v1/videos/captions",
        headers=headers,
        files={"video": f},
        data={"caption_template_id": "ctpl_DxflLOnuKkb198FNdI9E"}
    )
video_id = r.json()["video_id"]

Poll until complete:

# Poll every 10 seconds
r = requests.get(f"https://api.mirage.app/v1/videos/{video_id}", headers=headers)
status = r.json()["status"]  # QUEUED → PROCESSING → COMPLETE or FAILED

Download the captioned video:

r = requests.get(
    f"https://api.mirage.app/v1/videos/{video_id}/content",
    headers=headers,
    allow_redirects=True
)
with open("clip-FINAL.mp4", "wb") as f:
    f.write(r.content)

Video requirements for Captions.ai:

• Aspect ratio: 9:16 (Klap's output satisfies this)
• Max file size: 50 MB
• Max duration: 5 minutes
• Formats: MP4, MOV

Available caption templates (fetch full list via GET https://api.mirage.app/v1/videos/captions/templates):

Some popular templates:

Template	ID
Heat (default)	ctpl_DxflLOnuKkb198FNdI9E
Buzz	ctpl_yvE0ZnYzEj6ClCD2ee1f
Medusa	ctpl_yNnJyDLSH5oIouKdjQx2
Drive	ctpl_wR9PXfmxW1DFxEUuATFg
Magazine	ctpl_vrs1M2VrxvzQWNRypRvh
Energy	ctpl_oofP3mxbx8CaEPNYqnKD
Sirius	ctpl_miZu2nLWyP7X8oEAAHcM
Milky Way	ctpl_jcTmJGX77Uwz2AqLOX4S

Step 7: Generate Platform Captions

For each final clip, Claude writes platform-specific captions:

Instagram Reel:

• Hook line (first sentence people see)
• 2-3 sentences of context
• CTA (save, share, follow)
• 20-30 relevant hashtags
• Tone: professional but conversational

TikTok:

• Short, punchy caption (1-2 lines max)
• 5-8 hashtags
• Tone: casual, direct

YouTube Short:

• Title (under 60 characters, curiosity-driven)
• Description (2-3 sentences)
• Tags

LinkedIn (if applicable):

• Longer caption (3-5 sentences with a takeaway)
• Tone: professional, insight-driven

Step 8: Output

Save everything to the output directory:

<output-dir>/
  clip1-FINAL.mp4          # Ready-to-post clip
  clip2-FINAL.mp4
  clip3-FINAL.mp4
  captions.md              # All platform captions for each clip
  summary.md               # Overview: source video, clips made, scores, costs

Output specs:

• Format: MP4 (H.264)
• Resolution: 1080×1920 (9:16)
• Duration: 15-60 seconds per clip
• Audio: AAC

---

Workflow Summary

User provides video
      ↓
[ASK] "Do you want me to pick the best moments, or do you have specific timestamps?"
      ↓
Whisper transcribes locally (free)
      ↓
Claude scores moments on viral rubric (hook, quotability, emotion, self-contained, surprise)
      ↓
[ASK] "Here are the top N moments with scores. Approve, adjust, or add your own?"
      ↓
FFmpeg extracts raw clips (free)
      ↓
Klap reframes to 9:16 with speaker tracking (~$2/clip)
      ↓
Captions.ai adds animated captions (~$0.15/clip)
      ↓
Claude writes platform-specific captions
      ↓
Output: final clips + captions, ready to post

---

Known Limitations

1. yt-dlp may fail on some YouTube videos due to YouTube's evolving download restrictions. Install via brew install yt-dlp and keep updated. If download fails, user should download the video manually and provide the local file path.
2. Klap credit costs can add up at scale. Each clip costs ~76 credits (44 processing + 32 generation). Monitor credit balance before batch processing.
3. Captions.ai requires 9:16 input — always run Klap before Captions.ai, never the other way around.
4. Whisper base model is fast but may have transcription errors on technical terms, accents, or overlapping speech. Use whisper.load_model("medium") for better accuracy at the cost of slower transcription.
5. Viral scoring is heuristic — Claude's scoring is based on content patterns, not engagement data. Scores indicate relative quality within a video, not absolute viral potential.
6. Max 5 minutes per clip for Captions.ai, and 50MB file size limit. Klap has plan-based limits on video length (45 min to 3 hours depending on plan).
7. Processing time — Klap takes 2-5 minutes per clip, Captions.ai takes 1-2 minutes. A batch of 5 clips takes roughly 15-25 minutes total.

---

Environment Variables

Add these to your .env file:

KLAP_API_KEY=kak_xxxxx
CAPTIONS_AI_API_KEY=sk-xxxxx

No other API keys or local dependencies required. Whisper model downloads automatically on first run.