Gcp patterns

GCP Patterns

Cloud Run Deployment

Dockerfile for Cloud Run

FROM node:20-slim AS builder
WORKDIR /app
COPY package*.json ./
RUN npm ci --production=false
COPY . .
RUN npm run build

FROM node:20-slim
WORKDIR /app
RUN addgroup --system app && adduser --system --ingroup app app
COPY --from=builder /app/dist ./dist
COPY --from=builder /app/node_modules ./node_modules
COPY --from=builder /app/package.json ./
USER app
EXPOSE 8080
ENV PORT=8080 NODE_ENV=production
CMD ["node", "dist/server.js"]

Cloud Run Service YAML

apiVersion: serving.knative.dev/v1
kind: Service
metadata:
  name: order-service
  annotations:
    run.googleapis.com/launch-stage: GA
spec:
  template:
    metadata:
      annotations:
        autoscaling.knative.dev/minScale: "1"
        autoscaling.knative.dev/maxScale: "100"
        run.googleapis.com/cpu-throttling: "false"
        run.googleapis.com/startup-cpu-boost: "true"
    spec:
      containerConcurrency: 80
      timeoutSeconds: 300
      serviceAccountName: [email protected]
      containers:
        - image: gcr.io/project-id/order-service:latest
          ports:
            - containerPort: 8080
          resources:
            limits:
              cpu: "2"
              memory: 1Gi
          env:
            - name: DB_CONNECTION
              valueFrom:
                secretKeyRef:
                  key: latest
                  name: db-connection-string
          startupProbe:
            httpGet:
              path: /healthz
              port: 8080
            initialDelaySeconds: 5
            periodSeconds: 3

Deploy Command

gcloud run deploy order-service \
  --image gcr.io/$PROJECT_ID/order-service:$GIT_SHA \
  --region us-central1 \
  --service-account order-service@$PROJECT_ID.iam.gserviceaccount.com \
  --set-secrets "DB_URL=db-connection:latest" \
  --min-instances 1 \
  --max-instances 100 \
  --cpu 2 --memory 1Gi \
  --concurrency 80 \
  --no-allow-unauthenticated

BigQuery Optimization

-- Use partitioning and clustering
CREATE TABLE `project.dataset.events`
PARTITION BY DATE(event_timestamp)
CLUSTER BY user_id, event_type
AS SELECT * FROM `project.dataset.raw_events`;

-- Always filter on partition column
SELECT event_type, COUNT(*) as cnt
FROM `project.dataset.events`
WHERE event_timestamp BETWEEN '2025-01-01' AND '2025-01-31'
  AND event_type = 'purchase'
GROUP BY event_type;

-- Use approximate functions for large datasets
SELECT APPROX_COUNT_DISTINCT(user_id) as unique_users
FROM `project.dataset.events`
WHERE DATE(event_timestamp) = CURRENT_DATE();

-- Avoid SELECT * (scans all columns, costs more)
-- Use column selection and LIMIT for exploration

Pub/Sub Patterns

from google.cloud import pubsub_v1
from google.api_core import retry
import json

# Publisher with ordering and retry
publisher = pubsub_v1.PublisherClient()
topic_path = publisher.topic_path("project-id", "order-events")

def publish_event(event: dict, ordering_key: str = "") -> str:
    data = json.dumps(event).encode("utf-8")
    future = publisher.publish(
        topic_path,
        data,
        ordering_key=ordering_key,
        event_type=event["type"],
    )
    return future.result(timeout=30)

# Subscriber with exactly-once processing
subscriber = pubsub_v1.SubscriberClient()
subscription_path = subscriber.subscription_path("project-id", "order-events-sub")

def callback(message: pubsub_v1.types.PubsubMessage) -> None:
    try:
        event = json.loads(message.data.decode("utf-8"))
        idempotency_key = message.message_id

        if already_processed(idempotency_key):
            message.ack()
            return

        process_event(event)
        mark_processed(idempotency_key)
        message.ack()
    except Exception as e:
        logger.error(f"Failed to process message: {e}")
        message.nack()

subscriber.subscribe(subscription_path, callback=callback)

IAM Best Practices

Principles:
  - Least privilege: grant minimum permissions needed
  - Service accounts per service (not shared)
  - No user accounts in production workloads
  - Prefer predefined roles over primitive roles

Per-Service Pattern:
  order-service:
    roles:
      - roles/cloudsql.client          # DB access
      - roles/pubsub.publisher         # Publish events
      - roles/secretmanager.secretAccessor  # Read secrets
    # NOT: roles/editor (too broad)

Workload Identity (GKE):
  - Bind K8s SA to GCP SA
  - No key files, automatic credential rotation

Checklist

• Cloud Run services use dedicated service accounts
• Secrets stored in Secret Manager, not env vars
• BigQuery tables partitioned and clustered
• Pub/Sub subscribers implement idempotent processing
• Health check endpoints configured for all services
• Min instances set for latency-sensitive services
• IAM follows least privilege (no primitive roles)
• Cloud Armor WAF in front of public endpoints
• VPC connector for private resource access

Anti-Patterns

• Using default compute service account (overprivileged)
• SELECT * on BigQuery (scans all columns, high cost)
• Pub/Sub without dead letter queue (messages lost on repeated failure)
• Hardcoding project ID instead of using environment detection
• Not setting concurrency limits on Cloud Run (OOM under load)
• Using Cloud Run for long-running background jobs (use Cloud Tasks)
• Storing secrets in environment variables instead of Secret Manager