Gcp cloud storage

GCP Cloud Storage

Google Cloud Storage is a unified object storage service with global edge caching. It offers multiple storage classes (Standard, Nearline, Coldline, Archive) for cost optimization, strong consistency, and integration with all GCP services.

Core Concepts

• Bucket — globally unique container, scoped to a project and location
• Object — a file with metadata, identified by name (path) within a bucket
• Storage Class — Standard, Nearline (30d), Coldline (90d), Archive (365d)
• Signed URL — time-limited URL for authenticated access without credentials
• Lifecycle Rule — automatic actions based on age, class, or conditions
• IAM / ACL — access control at bucket and object level

Bucket Operations

# Create a bucket with location and default storage class
gcloud storage buckets create gs://my-app-assets-prod \
  --location=us-central1 \
  --default-storage-class=STANDARD \
  --uniform-bucket-level-access

# List buckets
gcloud storage ls

# Set bucket to public read (for static hosting)
gcloud storage buckets add-iam-policy-binding gs://my-app-assets-prod \
  --member=allUsers \
  --role=roles/storage.objectViewer

# Enable versioning
gcloud storage buckets update gs://my-app-assets-prod --versioning

Object Operations

# Upload a file
gcloud storage cp ./build/app.zip gs://my-app-assets-prod/releases/v1.2.0/app.zip

# Sync a directory
gcloud storage rsync ./dist gs://my-app-assets-prod/static/ \
  --delete-unmatched-destination-objects \
  --cache-control="public, max-age=86400"

# Copy between buckets
gcloud storage cp gs://source-bucket/data.csv gs://dest-bucket/data.csv

# List objects with prefix
gcloud storage ls gs://my-app-assets-prod/releases/ --recursive

# Remove objects
gcloud storage rm gs://my-app-assets-prod/old-file.txt

Signed URLs

# Generate signed URL for download (1 hour)
from google.cloud import storage
from datetime import timedelta

client = storage.Client()
bucket = client.bucket('my-app-assets-prod')
blob = bucket.blob('reports/q4.pdf')

url = blob.generate_signed_url(
    version='v4',
    expiration=timedelta(hours=1),
    method='GET'
)
print(f"Download URL: {url}")

# Generate signed URL for upload
url = blob.generate_signed_url(
    version='v4',
    expiration=timedelta(minutes=15),
    method='PUT',
    content_type='application/octet-stream'
)
print(f"Upload URL: {url}")
# Client uploads with: curl -X PUT -H "Content-Type: application/octet-stream" --data-binary @file "$url"

# Generate signed URL with gsutil
gcloud storage sign-url gs://my-app-assets-prod/reports/q4.pdf \
  --duration=1h \
  --private-key-file=service-account.json

Lifecycle Rules

// lifecycle-config.json — transition and expire objects automatically
{
  "rule": [
    {
      "action": {"type": "SetStorageClass", "storageClass": "NEARLINE"},
      "condition": {"age": 30, "matchesPrefix": ["logs/"]}
    },
    {
      "action": {"type": "SetStorageClass", "storageClass": "COLDLINE"},
      "condition": {"age": 90, "matchesPrefix": ["logs/"]}
    },
    {
      "action": {"type": "Delete"},
      "condition": {"age": 365, "matchesPrefix": ["logs/"]}
    },
    {
      "action": {"type": "AbortIncompleteMultipartUpload"},
      "condition": {"age": 7}
    },
    {
      "action": {"type": "Delete"},
      "condition": {"numNewerVersions": 3, "isLive": false}
    }
  ]
}

# Apply lifecycle rules
gcloud storage buckets update gs://my-app-assets-prod \
  --lifecycle-file=lifecycle-config.json

Static Website Hosting

# Configure bucket for static website
gcloud storage buckets update gs://my-app-website \
  --web-main-page-suffix=index.html \
  --web-not-found-page=404.html

# Upload website files with appropriate content types
gcloud storage cp -r ./build/* gs://my-app-website/ \
  --cache-control="public, max-age=3600"

Event Notifications

# Notify Pub/Sub when objects are created
gcloud storage buckets notifications create gs://my-app-assets-prod \
  --topic=storage-events \
  --event-types=OBJECT_FINALIZE \
  --object-prefix=uploads/

CORS Configuration

// cors-config.json — allow browser uploads
[
  {
    "origin": ["https://myapp.com"],
    "method": ["GET", "PUT", "POST"],
    "responseHeader": ["Content-Type"],
    "maxAgeSeconds": 3600
  }
]

# Apply CORS
gcloud storage buckets update gs://my-app-assets-prod --cors-file=cors-config.json

Access Control

# Grant a service account read access
gcloud storage buckets add-iam-policy-binding gs://my-app-assets-prod \
  --member=serviceAccount:[email protected] \
  --role=roles/storage.objectViewer

# Remove public access
gcloud storage buckets remove-iam-policy-binding gs://my-app-assets-prod \
  --member=allUsers \
  --role=roles/storage.objectViewer

Best Practices

• Enable uniform bucket-level access (disable ACLs) for simpler permissions
• Use lifecycle rules to automatically transition cold data to cheaper classes
• Generate signed URLs for temporary access instead of making objects public
• Enable versioning on buckets with critical data
• Use gcloud storage rsync for efficient directory synchronization
• Set appropriate Cache-Control headers for CDN and browser caching
• Use Pub/Sub notifications for event-driven processing of uploads
• Enable Object Versioning + lifecycle rules to limit stored versions