Compose cinematic mv

compose-cinematic-mv — music + intention → beat-synced cinematic MV

Turn a text intention (a mood / imagery / scene idea) plus music into a short vertical cinematic MV: design an original scene → first-frames with Seedream 4.0 (text-to-image) → motion with Kling (image-to-video) → cut the shots on the music's beats with ffmpeg. The aesthetic is modeled on Xiaohongshu creator Rainshae (lone figure in a vast landscape, cold cinematic mood, 3:4) — see reference/style-patterns.md.

The feeling comes from cuts riding the music, with short information-dense shots — not from pretty stills alone. So this skill is music-first and beat-synced.

Preconditions

• macOS with ffmpeg/ffprobe, curl, openssl, bc.
• Credentials as environment variables: ARK_API_KEY (Volcengine Ark / Seedream), KLING_AK, KLING_SK (Kling open platform). Keep them in a file outside this repo — e.g. ~/.config/secrets.env, chmod 600 — and source it before running: set -a; . ~/.config/secrets.env; set +a. Never paste keys into the chat.
• A working internet path to ark.cn-beijing.volces.com and api-beijing.klingai.com — see Network gotchas below; a local proxy will break these unless bypassed.

The workflow (music-first)

1. Design an original scene from the intention

From the user's intention, invent a NEW scene in the Rainshae idiom (title formula {年份}，你{第二人称处境}, lone figure / vast nature / strong mood / 3:4). Do not reproduce the reference creator's specific scenes (no "英国白崖", "长安", etc.). Read reference/style-patterns.md for the pattern, then create fresh.

2. Find matching music FIRST, then analyze its beats

Pick a background track that fits the scene's mood before planning shots. Reference the example videos for what fits: slow (~60–80 BPM), solo piano / ambient / strings, melancholic-serene ("深夜疗愈"), with phrase rise-and-fall (not a static drone).

• Source royalty-free / public-domain audio (e.g. archive.org public-domain piano — Satie Gymnopédie/Gnossienne, Chopin nocturnes, Debussy; or licensed ambient). archive.org direct files work; Pixabay blocks scraping (403).
• Avoid noisy recordings. archive.org "Great 78 Project" transfers (identifiers starting 78_ or containing gbia) have heavy surface hiss — denoise can't save them. Prefer modern/clean uploads. Verify the noise floor before committing: the quietest 0.5s RMS should be very low for a clean digital recording (≲ −60 dB); a noisy 78 sits around −40 dB. Check with: ffmpeg -v error -i x.mp3 -af "aformat=channel_layouts=mono,asetnsamples=22050,astats=metadata=1:reset=1,ametadata=print:key=lavfi.astats.Overall.RMS_level:file=nf.txt" -f null - ; grep RMS_level= nf.txt | sed -E 's/.*=//' | sort -n | head -1
• Download into the working dir, take a ~20–30s excerpt, normalize loudness, fade: ffmpeg -ss <start> -t 26 -i src.mp3 -af "loudnorm=I=-16:TP=-1.5,afade=t=in:st=0:d=1.2,afade=t=out:st=23.5:d=2.5" -ar 44100 -ac 2 excerpt.mp3
• Detect downbeats: scripts/find_beats.sh excerpt.mp3 → space-separated beat times. These are your cut points.

3. Plan shots ON the beats (short!)

• Short shots, ~2–3s each (5s reads as boring — each shot carries little info). Set the timeline cut points to a subset of the detected downbeats.
• One distinct generated video per shot — no reusing/re-cutting one clip into multiple shots (that was a cost-saving exception in the prototype, not the method).
• Mix shot types across the scene (wide establishing → medium figure → detail/atmosphere → closing long shot), camera moves slow.

4. Generate first-frames — Seedream, in parallel

For each shot write a model-ready English image prompt (subject, setting, era, framing, light, palette, mood, lens, film grain) ending in the right aspect. Generate concurrently (Seedream allows it):

set -a; . ~/.config/secrets.env; set +a
scripts/gen_image.sh "<prompt s1>" shot1.jpeg &
scripts/gen_image.sh "<prompt s2>" shot2.jpeg &
... ; wait

View each frame (Read tool) and regenerate any that miss before spending video credits.

5. Generate per-shot videos — Kling, parallel, short, fast poll

Submit up to 5 concurrent tasks; generate short single clips (duration 5, matching the ~2–3s shots — cheaper and fits fast cutting). Poll every ~6–8s.

scripts/kling.sh submit shot1.jpeg "<video prompt s1>" kling-v1-6 5   # -> task_id
# ...submit all, collect task_ids, then poll each until task_status=succeed
scripts/kling.sh poll <task_id>                                       # -> url when done
curl --noproxy '*' -o clip1.mp4 "<url>"                               # CDN needs --noproxy

Parse with grep (avoid python, see gotchas): status grep -oE '"task_status":"[a-z]+"'; url grep -oE '"url":"[^"]+"'.

6. Assemble beat-synced

Build plan.tsv (one line per shot, in order): clip<TAB>inpoint<TAB>duration, where the cumulative durations equal your chosen downbeat cut points. Then:

scripts/assemble.sh plan.tsv excerpt.mp3 out.mp4

Verify: probe duration, extract a couple frames at cut boundaries (Read) to confirm each cut changes the image.

7. QA the film — mandatory, before showing the user

You do the QA, not the user. Run every item below before showing a cut — all three failure classes have really happened:

• 物理/逻辑 bug：对每个 clip 抽 2–3 帧（开头/中间/结尾）用 Read 看。重点抓"运动载具窗外景色静止"这类矛盾（火车在开、窗外雪原不动 = 重做该 shot）、人脸/手部崩坏、水面倒影错误。
• 风格一致性：把所有 shot 的首帧并排看一遍，气质突兀的（如整组荒漠废墟里突然插两张壁画特写）换掉重生成，保持同一意象世界。
• 配乐：交付前完整听一遍 excerpt。有底噪/嘶声（RMS 检查漏网的）就换曲子，不要指望降噪。
• 剪辑：完整看一遍成片（或按节拍点抽帧），确认每个 cut 都换了画面、节奏踩在强拍上。

发现问题：只重做有问题的 shot（Kling 按条计费），别整片重跑。修完再交付，不要交付已知有 bug 的片子。

8. Deliver

Hand over out.mp4 together with the shot list and the track you used (title + source + license), so the user knows what they can publish.

Local overlay: if a file named SKILL.local.md exists next to this one, read it — that is where an individual user keeps their own private delivery conventions (where finished films get uploaded, which site to update, which host to deploy to). It is git-ignored and never part of this repo. Absent that file, just deliver the video.

Things that will bite you (all learned the hard way)

• Local proxy breaks the APIs. If the user runs ClashX/Clash (e.g. HTTPS_PROXY=127.0.0.1:7890), TLS to volces.com/klingai.com gets reset. Fix on their side: rule mode + DIRECT rules for volces.com and klingai.com. For downloads, always curl --noproxy '*' — result CDNs (kechuangai.com for Kling, Ark image CDN) are NOT covered by those rules and the proxy MITMs them (self-signed-cert / truncated downloads).
• Prefer bash + curl + openssl over python. In this sandbox python intermittently throws PermissionError: Operation not permitted on stdlib import (_path_importer_cache). All scripts here are python-free. (If you do use python for SSL, python.org's interpreter lacks CA certs → ssl.create_default_context(cafile="/etc/ssl/cert.pem").)
• Beat detection via ffmpeg, not librosa. numpy/librosa hit the same PermissionError; find_beats.sh uses an ffmpeg RMS envelope.
• Kling caps concurrency at 5. Submitting a 6th in-flight task returns 1303 parallel task over resource pack limit. Submit in batches of ≤5: poll+download the first batch, then submit the next.
• Account setup is the usual blocker, not code. Ark: the Seedream model must be activated in the Ark console (ModelNotOpen). Kling: the API resource pack must be purchased (1102 Account balance not enough) — distinct from klingai.com consumer credits.
• Quoting: prompts passed to the scripts must contain no double-quotes (JSON is built inline). Keep prompts to commas/letters.
• Older files can become unreadable. Files downloaded/created in earlier sessions may carry com.apple.quarantine/provenance and the sandbox denies access (Operation not permitted) — keep a run's working files together and don't depend on re-reading prior-session media.
• Model ids: Seedream doubao-seedream-4-0-250828; Kling kling-v1-6 (std). Endpoints: ark.cn-beijing.volces.com/api/v3/images/generations, api-beijing.klingai.com/v1/videos/image2video.

Files

• SKILL.md — this playbook.
• scripts/find_beats.sh — audio → downbeat times (ffmpeg RMS envelope).
• scripts/gen_image.sh — Seedream 4.0 text-to-image (curl); run in parallel.
• scripts/kling.sh — Kling image-to-video submit/poll (openssl JWT).
• scripts/assemble.sh — beat-synced assembler: (clip, in-point, duration) shots + music.
• reference/style-patterns.md — the Rainshae style规律 (title/visual/pacing/music) for creating original scenes.