Data migration

Data Migration Router

You plan and run the data layer of an EKS-to-GKE migration: managed databases, caches, queues, object storage, secrets. You produce per-system runbooks, then with explicit confirmation, drive the actual move.

Purpose

Convert "we have RDS Postgres, ElastiCache Redis, three S3 buckets, and MSK" into a moved, validated, observable GCP state — without dropping data, without unbounded downtime, and with a rollback path at every step. Routes database and data pipeline migration requests to specialized sub-procedures, ensuring all cross-cloud egress ([LFF-01]) and connectivity boundaries ([LFF-12], [LFF-13]) are validated prior to execution.

When to use this skill

• Phase 4 of a Portage migration.
• The user asks to "plan the data migration", "migrate RDS to Cloud SQL", "move S3 to GCS for the migration".

Prerequisites

• 01-discovery/inventory.json data dependencies section.
• 02-assessment/readiness-report.md with effort estimate per system.
• 03-landing-zone/plan.md with the data-prod project provisioned.
• Egress Budget Sign-Off ([LFF-01]): Section 7 of readiness-report.md must contain an executed model from tools/egress-estimator/egress_estimator.py.
• Deny-by-Default HITL Confirmation: Any mutation (dms start-replication-task, gcloud sql instances promote) requires interactive confirmation displaying command line and cost estimates.
• IAM permissions in source AWS for dms:* (where DMS is in scope) and read on source data systems.
• IAM permissions in GCP for the data services in scope (cloudsql.admin, redis.admin, etc.).

Procedure

Step 1 — Route to Sub-Procedure by System Type

For each data system in inventory.json, identify the store type and delegate to the corresponding sub-module:

• PostgreSQL / Cloud SQL: See postgres.md (includes public DNS/IP SSL allowlisting, [LFF-12], [LFF-13], [LFF-15]).
• MySQL / Cloud SQL: See mysql.md (includes GTID_MODE and binlog retention rules).
• Redis / Memorystore: See redis.md (includes RIOT-X seed and cache stampede strategies).
• S3 / GCS Storage Transfer: See s3.md (includes Storage Transfer Service incremental sync).
• Kafka / MSK Strimzi: See kafka.md (includes MirrorMaker 2 and consumer offset translation).
• Secrets Manager / Secret Manager: See secrets.md (includes External Secrets Operator bindings).
• DynamoDB / AlloyDB Re-platforming: See dynamodb.md (covers heterogeneous moves).

Step 2 — Co-existence connectivity

If the GKE workload needs to call back to the AWS data system during co-existence (and vice versa), plan:

• VPN or Cloud Interconnect between AWS VPC and GCP VPC.
• Allow-list updates to source data systems' security groups (allow GCP CIDRs).
• TLS / network policy parity.
• Bandwidth budget: project peak data movement at the planned co-existence window. Surface costs explicitly.

Step 3 — Per-system runbook with explicit gates

For each system, render a runbook (use templates/runbook-template.md) with:

• Pre-flight checklist.
• Cutover steps with expected commands and outputs.
• Validation gate after each step.
• Rollback procedure for each step (linking to rollback-playbook).
• Decommission steps and timing.

Step 4 — Drive the migration (with explicit confirmation)

For each system:

1. Surface the full runbook to the user. Ask explicit confirmation: "Begin replication for prod-payments-db? Estimated initial sync: 4–6 hours. Cost: $X."
2. On confirmation, execute step-by-step. After each step, run the validation gate. If a gate fails, stop, do not proceed.
3. Log every command executed and its output to 10-data-migration/<system>/execution.log.

The agent never destructively modifies the source. The agent never bypasses a failed gate.

Decision points

Decision	Default	When to deviate
Postgres / MySQL replication tool	DMS	pglogical native if DMS network constraints
Redis seed strategy	Rebuildable → empty target; durable → snapshot import	Online tail-based replication for true zero-downtime
S3 → GCS engine	Storage Transfer Service	gsutil/gcloud storage for one-shot, small datasets
MSK target	Confluent Cloud on GCP	Self-managed Kafka if cost or operator is fluent
DynamoDB target	Out of scope; re-platform project	(No default)
Soak duration on source	14 days read-only	7 days for non-prod, longer for tier-0

Outputs / Deliverables

10-data-migration/
├── plan.md                       # Index of all systems
├── <system-1>/
│   ├── runbook.md
│   ├── pre-flight-checks.md
│   ├── execution.log              # Filled in during execution
│   ├── validation-gates.md
│   └── rollback.md                # Links into rollback-playbook
├── <system-2>/
└── escalations.md

Validation

For each system, the cutover gate criteria must be defined and met before declaring complete:

• Postgres / MySQL: replication lag < target SLA for ≥ 5 min, row counts match for top 10 tables, sample row hashes match.
• Redis: target writeable + readable, app smoke tests pass on cached paths.
• S3 → GCS: object count matches, sample hashes match, lifecycle rules verified active on GCS.
• Kafka: consumer offset translation verified, MM2 lag ≈ 0, end-to-end produce → consume across both sides works.

Escalation triggers

• Heterogeneous data store moves (DynamoDB, Aurora→AlloyDB, Neptune, Redshift→BigQuery). Surface as scoping requirements.
• Datasets where the time-to-replicate exceeds the user's available window.
• Encryption key migrations that cannot reuse a CMEK approach (HSM-only key material). Surface for KMS planning.
• Any source data system with no acceptable cutover window AND no application-native replication. The migration stops until the user agrees to: (a) accept downtime, (b) rebuild downstream with new write-path, or (c) defer this system out of the migration.

Common pitfalls

• Egress during co-existence routinely runs 5–10× the unbudgeted estimate. Model it explicitly per-workload at assessment time and monitor near-real-time during execution. See LFF-01.
• Cross-cloud bulk transfer over open internet bottlenecks at default Linux TCP buffers. Tune net.ipv4.tcp_{r,w}mem before measuring throughput; otherwise you'll mis-time the cutover window. See LFF-11.
• Cloud SQL connection methods. Public IP is convenient but private IP via VPC peering or PSC is standard for prod. Use Cloud SQL Auth Proxy sidecar or Workload Identity-aware connector.

References

• Canonical sources: reference/sources.md.
• Database Migration Service overview.
• Storage Transfer Service.
• storage-translation — provides StorageClass, in-cluster volume strategies.
• traffic-cutover — sequences workload + data cutovers.
• rollback-playbook — gates failing during data migration cutover.
• docs/glossary.md — service map.