Media upload pipelines

Media Upload Pipelines

Instructions

Mobile uploads happen on flaky radios, at full battery cost, on large files. Never stream user media through your API pods. Offload to object storage via signed URLs and process asynchronously.

1. Architecture

 app -> POST /v1/uploads        (API pod -- tiny)
      <- { upload_url, object_key, upload_id }
 app -> PUT/PATCH upload_url    (object storage directly)
      <- 200 / 201
 app -> POST /v1/uploads/{id}/complete
                                -> triggers worker
                                        -> validate, scan, transcode
                                        -> emit "media.ready" event

• The API issues a short-lived signed URL.
• The client uploads directly to S3 / GCS / R2 / Azure Blob.
• A post-upload worker (triggered by object-event or by the /complete call) does the heavy lifting.

2. Signed URLs

• Short TTL (15 min typical, 1 h maximum).
• Scoped to a single object key and content-type.
• For small files (< 5 MB), a single PUT is enough.

// Go AWS SDK v2
func SignUpload(ctx context.Context, key, contentType string) (string, error) {
    ps := s3.NewPresignClient(s3client)
    out, err := ps.PresignPutObject(ctx, &s3.PutObjectInput{
        Bucket:      aws.String("uploads"),
        Key:         aws.String(key),
        ContentType: aws.String(contentType),
    }, s3.WithPresignExpires(15*time.Minute))
    if err != nil { return "", err }
    return out.URL, nil
}

3. Resumable Uploads

For files > 5 MB, single PUTs lose too much on failure. Use a resumable protocol:

• tus.io -- open protocol, works across object stores with a tus gateway; best client-library support (tus-java-client, TUSKit).
• S3 Multipart -- native on S3-compatible stores; client uploads parts, then sends a completion.
• GCS resumable -- simple session URL; PATCH with Content-Range.

tus upload (Kotlin, tus-android-client):

val upload = TusUpload(file).apply { metadata = mapOf("filename" to file.name) }
val uploader = TusUploader(tusClient, upload, upload.size)
while (uploader.uploadChunk() > -1) {
  progress.onProgress(uploader.offset, upload.size)
}
uploader.finish()

S3 multipart (Swift):

let init = try await s3.createMultipartUpload(bucket: bucket, key: key)
var parts: [CompletedPart] = []
for (i, chunk) in chunks.enumerated() {
  let etag = try await s3.uploadPart(bucket: bucket, key: key,
      uploadId: init.uploadId!, partNumber: i + 1, body: chunk)
  parts.append(.init(partNumber: i + 1, eTag: etag))
}
try await s3.completeMultipartUpload(bucket: bucket, key: key,
    uploadId: init.uploadId!, parts: parts)

4. Retry Semantics

• Single PUT: retry from zero on transient failure.
• Resumable: on failure, query the server for the current offset (HEAD for tus, ListParts for S3) and continue.
• Network change (Wi-Fi -> cellular): pause, confirm offset, resume. Do not restart blindly.
• Exponential backoff on 5xx and 429. Give up after ~5 attempts with a user-visible error.

5. Validation and Scanning

A post-upload worker must:

• Check file size and content-type match the signed URL parameters (prevent bypass).
• Sniff the actual file (magic bytes), not trust Content-Type from the client.
• Run a virus/malware scan for user-uploaded binaries (ClamAV, vendor service).
• Reject and delete the object on failure; emit a media.rejected event.

6. Derivatives

For images: generate thumbnails, WebP/AVIF variants (see image-resizing-service). For video: produce HLS/DASH renditions (see video-streaming). For audio: transcode to a common codec, normalize loudness if needed.

Derivatives go into a separate "processed" bucket with deterministic keys:

uploads/<user_id>/<upload_id>/original.<ext>
media/<media_id>/thumb_256.webp
media/<media_id>/hls/master.m3u8

7. Quotas and Abuse

• Per-user quota (daily and total), enforced before signing.
• Per-IP rate limits on /uploads to deter farming.
• Deny-list of repeated-offender hashes (perceptual hashes for media).
• Scan for CSAM / policy violations asynchronously; emit moderation events to review tooling.

8. API Endpoints

POST /v1/uploads
{ "content_type": "image/jpeg", "size": 2800000, "purpose": "avatar" }

-> { "upload_id": "up_01HX7...", "upload_url": "https://...", "expires_at": "...", "method": "PUT" }

POST /v1/uploads/up_01HX7.../complete
{ "etag": "abc" } // or multipart parts list

-> 202 { "media_id": "med_01HX7...", "status": "processing" }

GET /v1/media/med_01HX7...
-> { "id": "...", "status": "ready", "url": "https://cdn...", "variants": { "thumb_256": "..." } }

9. Client Consumption

• Show progress tied to uploaded bytes, not naive "pending" / "done".
• Upload in the background using platform-appropriate APIs: URLSession background configuration on iOS; WorkManager with expedited constraints on Android.
• Pause on low battery / metered network if user setting allows.
• Persist the upload_id so the client can resume across app restarts.

10. Observability

Track per stage: signing latency, upload duration, upload failure reasons (network vs 4xx vs 5xx), processing queue lag, end-to-end latency (tap-to-ready).

Checklist

• Uploads bypass API pods via signed URLs.
• Resumable protocol (tus / multipart) used for files > 5 MB.
• Signed URL TTL ≤ 1 h; scoped to one key + content-type.
• Post-upload worker validates size, sniffs content, and scans.
• Derivatives written to a separate bucket with deterministic keys.
• Per-user quota and per-IP rate limits enforced before signing.
• Client uses background upload APIs and persists upload_id for resume.
• End-to-end latency metrics tracked and alerted.