agentsclimarketplace

Multimodal extraction

Skill biyearly-mesothelioma790/skills/multimodal-extraction

Organize reusable AI agent skills for Claude Code, Codex, and Cursor with focused workflows, scripts, and fast task selection

Install
npx -y skills add biyearly-mesothelioma790/skills --skill multimodal-extraction

Assembled from the repository path, not quoted from the project. Check it against their README if it does not work.

2 things to look at

  • no licenseNo license file was found in the repository. Code published without one is not open source by default, so using it at work is a question for whoever answers licensing questions where you are.
  • 0 stars0 stars. Stars are a popularity signal and not a quality one, but at this level it is likely that nobody has read this closely except its author, and you would be relying on your own review.

What its author says it does

Copied from the file, not written here

Given a local video or video URL, downloads the media if needed, extracts slide frames and key moments, transcribes the audio, and writes a Markdown timeline that interleaves screenshots with the transcript at the associated timestamps. Use when asked to turn a video into a multimodal notes file, slide-synced transcript, screenshot-enhanced transcript, or talk recap with images.

SKILL.md

3.6 KB, as published. Nobody here has run it

Multimodal Extraction

Overview

This skill composes the existing video workflows into one artifact:

  • download-video for URL inputs
  • thumbnail-extraction for slide frames and key screenshots
  • transcribe-anything for transcript strategy

The implementation is intentionally speed-first:

  1. Download only when the input is a URL
  2. Reuse the fast slide/key-frame heuristics from thumbnail-extraction
  3. Use local whisper JSON output for timestamped transcript segments
  4. Merge everything into one Markdown timeline with relative image links

When To Use

  • "Turn this talk into multimodal notes"
  • "Make me a markdown transcript with screenshots"
  • "Extract slides and transcript together"
  • "Build a recap doc from this video"
  • "Given this YouTube URL, produce a slide-synced transcript"

Requirements

brew install ffmpeg yt-dlp
pip3 install --break-system-packages openai-whisper

The following existing local script is reused:

  • ../thumbnail-extraction/thumbnail_extractor.py

Command

python3 multimodal_extract.py <video_or_url> [output_dir] [--language en] [--whisper-model turbo] [--top-n 4]

What It Does

Step 1: Resolve the Source

  • If the input is a local file, use it directly
  • If the input starts with http:// or https://, download it first with yt-dlp
  • For YouTube URLs, direct yt-dlp is usually enough
  • For trickier hosted pages, this skill follows the same practical intent as download-video: get a usable local file first

Step 2: Extract Visual Anchors

Run:

python3 ../thumbnail-extraction/thumbnail_extractor.py "$VIDEO" "$OUTPUT/visuals" 4 --extract-slides

This produces:

  • top thumbnail candidates in the root of visuals/
  • slide images in visuals/slides/
  • manifests with timestamps

Step 3: Transcribe

Extract normalized mono 16k audio:

ffmpeg -y -i "$VIDEO" -vn -ac 1 -ar 16000 -acodec pcm_s16le \
  -af "highpass=f=80,lowpass=f=8000,loudnorm=I=-16:TP=-1.5:LRA=11" \
  "$OUTPUT/audio/source_preprocessed.wav"

Then transcribe with Whisper:

whisper "$OUTPUT/audio/source_preprocessed.wav" \
  --model turbo \
  --language en \
  --word_timestamps True \
  --condition_on_previous_text False \
  --output_format json \
  --output_dir "$OUTPUT/transcript"

Step 4: Merge into Markdown

The script:

  • reads slide and thumbnail manifests
  • reads Whisper transcript segments
  • sorts all visual anchors by timestamp
  • groups transcript text between successive visual anchors
  • writes multimodal_timeline.md with:
    • section timestamp
    • associated image(s)
    • transcript span for that interval

Output

output_dir/
  source/
  visuals/
  audio/
  transcript/
  multimodal_timeline.md

Design Principle

The goal is total end-to-end extraction speed.

That means:

  • heuristics first
  • local transcript by default
  • no VLM in the common path
  • only enough structure to make the Markdown artifact useful immediately

Future Extensions

  • add backend switching for transcribe-anything
  • add deck-aware slide labeling when a source deck exists
  • add speaker diarization sections
  • add chaptering or summary generation on top of the Markdown timeline

Keep looking

Skills are one crate of 328,083. Ordering is by how many stacks a row turns up in, so the top of any crate is what has actually been picked rather than what has the most stars.