Video analyzer repo

Video Analyzer Skill

Converts a video file into a complete, structured report (Markdown, JSON and PDF) that captures everything an AI or human needs to understand the video without watching it. Includes audio transcription (Whisper), visual frame analysis (Claude Vision), and optional scene clustering to aid frame selection.

---

Installation

Claude Code / Cowork (full execution)

pip install openai-whisper opencv-python-headless numpy Pillow \
    --break-system-packages -q 2>/dev/null || true

# PDF renderer — install one (weasyprint preferred, reportlab as fallback):
pip install weasyprint      # requires GTK3 on Linux/macOS
pip install reportlab       # pure Python, works everywhere

Running build_json_report.py automatically generates MD + JSON + PDF in a single invocation. No separate command is needed for the PDF.

Claude.ai Desktop / Web (read-only mode)

Upload SKILL.md as an instruction file in a Claude Project. The skill will describe the analysis process and interpret pre-generated JSON/MD reports, but cannot execute ffmpeg, Whisper, or the Python scripts without Claude Code or Cowork. For full execution (frame extraction, transcription, PDF generation) use Claude Code.

Customising the PDF template

The CSS and HTML layout live entirely inside scripts/build_pdf_report.py — no external stylesheets or assets. To customise colours, fonts, or sections:

1. Edit the _CSS string constant near the top of the file.
2. Edit the _build_html() function for structural changes (sections, order).
3. Edit _build_pdf_reportlab() for the reportlab fallback layout.

The generate_pdf(json_path, frames_dir, output_dir) function is the stable public API — call it from any script without touching the internals.

Token cost estimates (choose your mode):

• Character-counting pass only (5–8 frames): ~8–12k vision tokens
• Full frame-by-frame timeline (up to 30 frames): ~40k+ vision tokens

---

Step 0 — Find the video file

ls /mnt/user-data/uploads/

If no video is present, ask the user to upload via the 📎 icon.

---

Step 1 — Install dependencies & run pre-processing

pip3 install openai-whisper opencv-python-headless numpy Pillow \
    --break-system-packages -q 2>/dev/null || true

python3 /home/claude/video-analyzer/scripts/process_video.py \
    "<VIDEO_PATH>" \
    --output-dir /tmp/video_work \
    --model large-v3-turbo

Frame sampling is automatic and adaptive — no configuration required. process_video.py scans the video at 0.25 s intervals, measures visual change with cv2.absdiff, and extracts a frame only when the scene changes significantly, or at least every 30 s on static content (floor), capped at 60 frames (ceiling). Frames are resized so the long edge is at most 1568 px (aspect ratio preserved).

Parameter guide:

Parameter	Default	When to change
--fps	(adaptive)	Pass a value (e.g. 0.5) only to force legacy fixed-fps extraction. A warning is logged and adaptive sampling is disabled for that run.
--max-frames	30	Applies only when --fps is explicit. Adaptive mode uses its own internal cap (60).
--model	large-v3-turbo	Whisper model: tiny/base/small/medium/large-v3-turbo

---

Step 2 — (Optional) Run visual scene clustering

This step groups frames by color/composition similarity to help you pick representative frames for Claude Vision. It is not a character detector — it uses HSV histograms, not face recognition. Skip this step if you prefer to select frames manually.

python3 /home/claude/video-analyzer/scripts/cluster_frames.py \
    --frames-dir /tmp/video_work/frames \
    --output /tmp/video_work/clusters.json \
    --manifest /tmp/video_work/manifest.json \
    --similarity-threshold 0.72

The output clusters.json contains visual_cluster_count (a rough grouping estimate) and a representative_frame path per cluster. Use these paths to choose the 5–8 frames for the character-counting pass below.

Tuning --similarity-threshold:

• Too many clusters (>8 for a simple video): raise to 0.80
• Too few clusters (everything merged): lower to 0.60

---

Step 3 — Read metadata, transcript, and cluster data

cat /tmp/video_work/metadata.json
cat /tmp/video_work/transcript.txt
cat /tmp/video_work/transcript_segments.json   # confidence scores per segment (avg_logprob, confidence, low_confidence)
cat /tmp/video_work/clusters.json   # omit if Step 2 was skipped
cat /tmp/video_work/manifest.json

---

Step 4 — Identify characters with Claude Vision (character-counting pass)

Pick 5–8 well-distributed frames from the manifest (or use the representative_frame paths from clusters.json). Inspect each with the view tool.

view: /tmp/video_work/frames/frame_0001.jpg
view: /tmp/video_work/frames/frame_0009.jpg
…

For each frame, note every distinct person or character visible: appearance, clothing, setting. Then synthesize across all frames:

• How many distinct characters appear in total?
• Give each a short label and description (e.g. "Character 1 — Gioggia: blonde, white suit, office setting").

This count and identification come from your visual judgment, not from the clustering script. The skill targets AI-generated and stylized video where geometric face detectors are unreliable; a vision model reasoning about "distinct characters" is more appropriate.

Record your findings:

Character 1: <name/label> — <description>
Character 2: <name/label> — <description>
…

---

Step 5 — Full visual timeline (optional, ~40k+ tokens)

If the user wants a frame-by-frame timeline, sweep through all frames in the manifest. Group near-identical consecutive frames and write a 2–4 sentence description per distinct scene:

[00:00:00] <who is present, setting, action, any on-screen text>
[00:00:05] <…>
…

Skip this step for a summary-only report to keep token cost low.

---

Step 6 — Build the JSON report

python3 /home/claude/video-analyzer/scripts/build_json_report.py \
    --work-dir /tmp/video_work \
    --output /mnt/user-data/outputs/<basename>_report.json \
    --visual-timeline '<JSON_ARRAY>' \
    --character-names '<JSON_OBJECT>' \
    --summary "<SUMMARY_TEXT>" \
    --key-observations '<JSON_ARRAY>'

Example values:

--visual-timeline '[{"timestamp":"00:00:00","description":"Donna bionda..."}]'
--character-names '{"1":"Gioggia","2":"Antoni","3":"Marina","4":"Anziano"}'
--summary "Video di satira politica AI-generated..."
--key-observations '["Prodotto con SHOWRUNNER.XYZ","4 personaggi distinti"]'

--character-names is the output of your Claude Vision identification pass (Step 4). If you skipped Step 4 (summary-only), pass {}.

---

Step 7 — Build the Markdown report

Save to /mnt/user-data/outputs/<basename>_report.md using this template.

Confidence marking rules (apply when writing Section 3):

• Read audio.transcript_segments from the JSON report (field low_confidence is bool or null).
• Count N = total segments, M = segments where low_confidence == true.
• For each segment:
  • low_confidence: true → render the entire line in italic, prepend ⚠️ [trascrizione incerta]
  • low_confidence: false → render normally
  • low_confidence: null or field absent → render normally but append [confidenza non disponibile]
• Never alter or omit the transcribed text — marking is purely presentational.

# 📹 Video Report: <filename>
> Auto-generato dalla skill video-analyzer.

> **Trascrizione: N segmenti totali, M a bassa confidenza (da verificare manualmente)**
>
> **Nota sulle sorgenti:** Il testo in sezione 3 (🎤 audio) proviene dalla trascrizione
> Whisper ed è verificabile riascoltando il video. Le descrizioni in sezione 4
> (👁 visione) sono generate da Claude Vision su singoli fotogrammi: inferenza automatica,
> può contenere errori.

## 1. Metadata
| Proprietà | Valore |
|-----------|--------|
| Filename | `...` |
| Durata | `HH:MM:SS` |
| Risoluzione | `WxH` |
| Video codec | `...` |
| Audio codec | `...` |
| Dimensione | `N MB` |
| FPS | `N` |
| Bitrate | `N kbps` |
| Whisper model | `<metadata.whisper_model>` |
| Vision model | `<metadata.vision_model>` |

## 2. Personaggi Identificati
*(N personaggi distinti identificati da Claude Vision)*

| ID | Nome/Label | Descrizione |
|----|-----------|-------------|
| 1  | ...        | ...         |

## 3. Trascrizione Audio — 🎤 sorgente: Whisper (verificabile riascoltando il video)
*(N segmenti totali · M a bassa confidenza · Whisper `<metadata.whisper_model>` · Lingua: `<audio.language_note>`)*

> **Trascrizione: N segmenti totali, M a bassa confidenza (da verificare manualmente) · Lingua: `<audio.language_note>`**
>
> *(Se `audio.language_detection_prob` < 0.60, aggiungere: "⚠️ Rilevamento lingua incerto — se il video contiene più lingue o audio poco chiaro, la trascrizione può contenere errori sistematici.")*

[HH:MM:SS] Esempio di segmento normale con confidenza alta.
*⚠️ [trascrizione incerta] [HH:MM:SS] Esempio di segmento con low_confidence: true — testo originale invariato.*
[HH:MM:SS] Esempio di segmento senza campo confidence. [confidenza non disponibile]

## 4. Timeline Visiva — 👁 sorgente: Claude Vision (interpretazione automatica di fotogrammi)

> **Nota:** Le descrizioni seguenti sono generate da Claude Vision analizzando singoli
> fotogrammi estratti dal video. Si tratta di inferenza automatica: può contenere errori
> di identificazione, attribuzione o interpretazione. Non equivale alla trascrizione audio.

### [HH:MM:SS] — Scena N
> 👁 [descrizione automatica immagine] <descrizione 2-4 frasi di cosa Claude Vision vede nel frame>

## 5. Sommario AI
<150–300 parole>

## 6. Osservazioni Chiave
- ...

---

Step 8 — Present both files

present_files([
    "/mnt/user-data/outputs/<basename>_report.md",
    "/mnt/user-data/outputs/<basename>_report.json"
])

---

Tips & edge cases

• No audio: set transcript to "No audio track detected."
• Screen recordings: pay attention to UI, on-screen text, window titles
• Fast-cut video: adaptive sampling handles this automatically; no flag needed
• Very long (>30 min) static video: the 30 s floor guarantees coverage; no flag needed
• Override sampling: pass --fps 0.5 to revert to legacy fixed-fps mode (e.g. for debugging)
• Foreign audio: Whisper is multilingual, no configuration needed
• AI-generated / animated video: face detectors won't work; rely entirely on Claude Vision for character identification — this is the expected use case
• Skip clustering entirely: just pick frames evenly spaced from the manifest for the character-counting pass

---

Scripts

Script	Role
process_video.py	Pre-processing: metadata → scene detection → audio → Whisper → intermediate_nodes.json
cluster_frames.py	Optional: groups scene frames by HSV similarity for representative frame selection
build_json_report.py	Semantic compression + global_index + JSON report; auto-calls build_pdf_report.py
build_pdf_report.py	PDF generator (weasyprint → reportlab fallback). Entry point: generate_pdf(json, frames_dir, out_dir)

Intermediate files

File	Description
metadata.json	ffprobe metadata (duration, codec, resolution, …)
manifest.json	Frame paths and scene-change timestamps
transcript.txt	Whisper transcript with [HH:MM:SS] markers
clusters.json	Visual clusters from cluster_frames.py (optional)
intermediate_nodes.json	Formato pivot interno da cui vengono generati tutti gli output. Non eliminare tra le esecuzioni.
transcript_segments.json	Segmenti Whisper strutturati con punteggi di confidenza: avg_logprob (raw Whisper), no_speech_prob, confidence (normalizzata 0–1), low_confidence (bool). Letto da build_json_report.py per arricchire audio.transcript_segments nel JSON finale. Valori: confidence vicino a 1.0 = trascrizione affidabile; low_confidence: true = verificare manualmente.