Avfoundation media pipeline workflow

AVFoundation Media Pipeline Workflow

Purpose

Guide AVFoundation capture, playback, asset loading, reader, writer, export, and sample-buffer pipeline design and repair. This skill owns media-pipeline shape, typed AVFoundation surface choice, and back-pressure policy, while leaving app audio-session policy, engine graphs, Core Media timing internals, and Xcode execution to their owning skills.

When To Use

• Use this skill for AVCaptureSession, capture inputs and outputs, capture queues, AVPlayer, AVPlayerItem, AVAsset, AVURLAsset, AVAsyncProperty, async asset loading, AVAssetReader, AVAssetWriter, export, transcode, and sample-buffer append loops.
• Use this skill when existing code blocks the main thread, uses deprecated AVAsset synchronous properties, misuses loadValuesAsynchronously(forKeys:), appends sample buffers without back-pressure, confuses capture authorization with microphone permission, or mixes capture, UI, writer, and player ownership in one object.
• Recommend avfaudio-session-workflow when the primary problem is app audio intent, routes, interruptions, or microphone permission.
• Recommend coremedia-timing-samplebuffer-workflow when the primary issue is sample-buffer timestamps, format descriptions, timebases, clocks, or synchronization.
• Recommend camera-capture-depth-workflow when the primary issue is camera discovery, device controls, formats, rotation, photo capture, depth, calibration, synchronized camera outputs, or computational capture.

Single-Path Workflow

1. Classify the media pipeline:
   • capture setup
   • player or playback state
   • asset inspection or async loading
   • reader or writer loop
   • export or transcode
   • sample-buffer append or back-pressure
   • repair of an existing implementation
2. Apply the Apple docs gate:
   • read current AVFoundation documentation for the pipeline surface
   • state the documented behavior relied on before recommending changes
   • apply ../../shared/references/apple-media-type-ownership.md before introducing custom media wrappers, raw numeric timing, strings, or dictionaries
3. Choose the pipeline owner and queues:
   • capture-session configuration owner
   • serial capture session queue
   • player item and UI update boundary
   • async asset loading boundary
   • reader and writer queue
   • cancellation and teardown path
4. Repair common failure modes:
   • AVCaptureSession.startRunning() on the main queue
   • deprecated synchronous AVAsset property reads in Swift clients
   • callback-based asset loading left in otherwise async Swift code
   • writer loops that ignore isReadyForMoreMediaData
   • raw strings, dictionaries, or custom structs replacing AVAsset, AVAssetTrack, AVPlayerItem, AVCaptureConnection, AVAssetReaderOutput, AVAssetWriterInput, CMSampleBuffer, or writer receiver types without a documented boundary reason
   • capture authorization, recording permission, and Info.plist gates collapsed together
   • sample-buffer timing issues that need Core Media handoff
5. Return one recommendation with:
   • pipeline class
   • documented Apple behavior relied on
   • owner, queue, and async-loading plan
   • back-pressure and cancellation plan
   • repair findings
   • validation and handoff expectation

Inputs

• request: optional free-text media task.
• pipeline_goal: optional goal such as capture, playback, asset-loading, reader, writer, export, transcode, sample-buffer-append, or repair.
• platform_context: optional platform emphasis such as ios, macos, or mixed-apple.
• Defaults:
  • docs-first guidance always applies
  • prefer Apple and Swift media types unless ../../shared/references/apple-media-type-ownership.md identifies a concrete escape hatch
  • prefer Swift concurrency asset loading for modern Swift clients
  • keep blocking session and media work off the main queue

Outputs

• status
  • success: a media pipeline recommendation or repair path is ready
  • handoff: another Apple Dev skill owns the next step
  • blocked: pipeline ownership or runtime evidence is too unclear
• path_type
  • primary: current AVFoundation docs cover the recommendation
  • fallback: a lower-level timing or audio repair workflow is needed
• output
  • resolved pipeline class
  • documented Apple behavior relied on
  • owner, queue, loading, and back-pressure plan
  • repair findings
  • validation and handoff expectation

Guards and Stop Conditions

• Do not run capture sessions, media loading, export, or writer loops on the main queue when docs identify the operation as blocking or asynchronous.
• Do not replace AVFoundation or Core Media pipeline types with custom wrappers, strings, dictionaries, or raw numeric timing unless the conversion boundary and lost media information are explicit.
• Do not claim capture, playback, export, camera, microphone, or route behavior is verified without runtime evidence.
• Do not silently absorb Core Media timestamp, sample-buffer readiness, or timebase repair.
• Do not collapse UI state, capture configuration, writer loops, and player ownership into one object unless the code is deliberately tiny and no persistent boundary is needed.
• Stop with blocked when the requested behavior depends on unavailable camera, microphone, file, route, or device evidence.

Fallbacks and Handoffs

• Recommend avfaudio-session-workflow for app audio intent, microphone permission, interruptions, routes, or capture-session audio-session interaction.
• Recommend camera-capture-depth-workflow for camera devices, controls, photo features, depth, calibration, synchronized outputs, and capability diagnostics.
• Recommend coremedia-timing-samplebuffer-workflow for CMTime, format descriptions, sample-buffer attachments, dropped buffers, or synchronization.
• Recommend video-codec-processing-workflow for low-level VideoToolbox compression/decompression, codec properties, hardware policy, pixel-buffer pools, and per-frame callbacks.
• Recommend photos-library-editing-workflow for PhotosUI selection, PhotoKit asset/resource requests, library saves, and content editing.
• Recommend avaudio-engine-workflow for engine graph, tap, or audio processing work.
• Recommend xcode-build-run-workflow for build, run, target, entitlement, Info.plist, or device follow-through.
• Recommend xcode-testing-workflow for repeatable media tests, fixtures, or runtime verification planning.
• Recommend explore-apple-swift-docs when docs lookup is the real need.

Customization

Use references/customization-flow.md.

scripts/customization_config.py exists to preserve the repo-wide customization-file contract, but this workflow defines no runtime-enforced knobs.

References

Workflow References

• references/media-pipeline-and-repair.md
• references/async-loading-and-backpressure.md
• references/customization-flow.md

Support References

• Use ../../shared/references/apple-media-type-ownership.md for the shared Apple media type and framework-selection contract.
• Recommend references/snippets/apple-xcode-project-core.md when the user needs reusable Xcode-project baseline policy for media apps.

Script Inventory

• scripts/customization_config.py