Remote logging

Remote Log Shipping on Mobile

Instructions

Mobile clients are unreliable log sources: networks flap, batteries die, users background the app. Ship logs defensively: batch, sample, redact, and never block the UI thread.

1. Pick a Transport

• OTel log exporter (OTLP/HTTP) to a collector you own. Most flexible; collector handles fan-out to Loki / Datadog / Elastic.
• Vendor SDK (Datadog Logs SDK, Sentry logs, Grafana Faro). Less config, more lock-in.
• Raw HTTPS POST to your own endpoint. Only reasonable if you already run an ingest service.

Prefer OTLP so you can re-route without touching the client. Send via HTTP/protobuf, not gRPC (better behavior on constrained mobile networks).

2. Batch and Schedule

• Batch size: up to 100 records or 64 KB, whichever first.
• Flush interval: every 30 s in foreground, 120 s in background (iOS background URLSession / Android WorkManager).
• Never flush on UI thread. All work on a dedicated dispatcher (Kotlin) / queue (Swift) / isolate (Dart) / native bridge async task (RN).
• Coalesce: dedupe identical log lines within a flush window; increment attrs.count instead of sending N copies.

3. Sampling

• Sample by logger name and level: 100% of error/fatal, 10-20% of info, 1-5% of debug, 0% of trace.
• Per-user-per-day budget: cap at 5000 log records / user / day; after that, keep errors only.
• Always keep 100% of logs that share a trace with a span marked error=true (tail sampling at the collector if possible).

fun shouldKeep(entry: LogEntry): Boolean = when (entry.level) {
    "error", "fatal" -> true
    "warn"           -> Random.nextDouble() < 0.5
    "info"           -> Random.nextDouble() < 0.15
    "debug"          -> Random.nextDouble() < 0.02
    else             -> false
}

Log the sample rate alongside the record as attrs.sample_rate so the backend can re-inflate counts.

4. Offline Buffering

• Persist unsent batches to a bounded on-disk ring buffer (e.g. SQLite table, Dart hive, iOS file-based queue).
• Cap the buffer (e.g. 10 MB / 50 000 records). Drop oldest on overflow and emit a meta-log remote_log_overflow.
• Encrypt the buffer at rest if logs can contain anything sensitive: EncryptedSharedPreferences / DataStore with MasterKey on Android, Keychain-wrapped key on iOS.
• Clear the buffer on sign-out if logs can be tied to the signed-out user.

5. Backpressure

• If the backend returns 429 or 5xx, apply exponential backoff (base 2 s, max 5 min, full jitter).
• If backoff count exceeds a threshold, pause shipping and mark a Session.logsPaused = true so feature code can skip expensive logs.
• Do not retry forever on 400/401 -- drop the batch and alert in-app telemetry (see metric-alerts).

6. PII and Redaction in Transit

• Re-run the redactor at the shipper boundary even if already run at emit -- defense in depth.
• Strip HTTP Authorization headers, cookies, and body payloads from instrumented network logs.
• Redact stack trace filenames that contain user directories (/Users/alice/...). Replace with ~/.
• TLS is mandatory. Pin certificates for your own ingest endpoint; do not pin to vendor endpoints that rotate certs.

7. Example: Android OTel Log Exporter

val logExporter = OtlpHttpLogRecordExporter.builder()
    .setEndpoint("https://otel.example.com/v1/logs")
    .addHeader("Authorization", "Bearer ${BuildConfig.OTEL_TOKEN}")
    .setTimeout(Duration.ofSeconds(10))
    .build()

val logProvider = SdkLoggerProvider.builder()
    .addLogRecordProcessor(
        BatchLogRecordProcessor.builder(logExporter)
            .setMaxExportBatchSize(100)
            .setScheduleDelay(Duration.ofSeconds(30))
            .setMaxQueueSize(2048)
            .build()
    )
    .setResource(Resource.create(Attributes.of(
        AttributeKey.stringKey("service.name"), "my-app",
        AttributeKey.stringKey("service.version"), BuildConfig.VERSION_NAME,
        AttributeKey.stringKey("deployment.environment"), BuildConfig.FLAVOR
    )))
    .build()

8. iOS Background URLSession

let config = URLSessionConfiguration.background(withIdentifier: "logs.upload")
config.sessionSendsLaunchEvents = false
config.isDiscretionary = true
config.allowsCellularAccess = true
logsSession = URLSession(configuration: config, delegate: self, delegateQueue: nil)

iOS will schedule uploads when the device is charging and on Wi-Fi if isDiscretionary = true -- good for non-urgent logs, bad for error logs. Keep a parallel foreground session for errors.

9. Cost and Cardinality

• Cardinality killers: adding user_id_hash, device_model, or session_id as indexed attributes. Keep them as unindexed fields.
• Do not log once per frame, once per scroll event, or once per network byte.
• Review top loggers by ingested volume weekly; anything > 1% of total volume needs justification.

10. Verification

• Unit test: feeding 10k records generates <= 100 batches of <= 64 KB.
• Integration test: airplane mode for 5 minutes then reconnect drains the queue without loss up to the buffer cap.
• Chaos test: return 503 for 60 seconds; shipper backs off and eventually succeeds.

Checklist

• All shipping runs off the UI thread on a dedicated worker.
• Batches are bounded by size and time; flush interval differs foreground vs background.
• Sampling by level + per-user-per-day budget is implemented; sample_rate is attached.
• Offline buffer is bounded, encrypted if sensitive, and cleared on sign-out.
• Backoff on 429/5xx with jitter; 4xx drops the batch.
• Redaction runs both at emit and at the shipper boundary.
• TLS enforced; cert pinning on owned endpoints.
• Cost/cardinality review is part of the release checklist.