diff --git a/PRIVATE_IP_SETUP.md b/PRIVATE_IP_SETUP.md new file mode 100644 index 00000000..85cd7c07 --- /dev/null +++ b/PRIVATE_IP_SETUP.md @@ -0,0 +1,160 @@ +# Google Cloud Creative Studio: Private IP Deployment Guide + +This guide details how to deploy Creative Studio with a **Private IP only** Cloud SQL database and configure the **Cloud Run** backend to access it securely using **Direct VPC egress**. + +Because the database does not expose a public IP, the initial setup is split into a **two-step lifecycle** to comply with network security boundaries. + +--- + +## Architecture Overview +1. **Private Database:** The Cloud SQL Postgres database has public access disabled (`ipv4_enabled = false`) and uses VPC Peering (`servicenetworking.googleapis.com`) to receive connections. +2. **Serverless Egress:** Cloud Run backend service is connected directly to the VPC subnet using Direct VPC egress (`egress = "ALL_TRAFFIC"`), allowing it to query the database internally. +3. **Secure Seeding:** Database seeding is performed using a temporary jump-box VM residing inside the VPC to securely load templates and assets. + +--- + +## Setup Lifecycle (Step-by-Step) + +### Phase 1: Deploy Infrastructure +Run the bootstrap script from your standard **Google Cloud Shell** session to deploy the network, database, secrets, and Cloud Build triggers: + +```bash +curl -sSL https://raw.githubusercontent.com/PKAgarwal157/gcc-creative-studio/private-ip-cloudsql/bootstrap.sh | bash +``` +* When prompted for your fork URL, enter: `https://github.com/PKAgarwal157/gcc-creative-studio.git` +* When prompted for the branch, enter: `private-ip-cloudsql` +* *Note: This script will complete successfully but will skip the database seeding step, which must be run internally inside the VPC.* + +--- + +### Phase 2: Grant Seeding Permissions +The temporary setup VM will run as the project's **Default Compute Engine Service Account**. Before creating the VM, run these commands in your **Cloud Shell** to grant the service account access to Secret Manager, Cloud SQL, and the Asset Storage bucket: + +```bash +# Get your project details +export PROJECT_ID=$(gcloud config get project) +export PROJECT_NUM=$(gcloud projects describe $PROJECT_ID --format="value(projectNumber)") +export COMPUTE_SA="${PROJECT_NUM}-compute@developer.gserviceaccount.com" + +# 1. Grant Secret Manager Access (to retrieve DB password) +gcloud secrets add-iam-policy-binding "creative-studio-db-password" \ + --role="roles/secretmanager.secretAccessor" \ + --member="serviceAccount:$COMPUTE_SA" \ + --project="$PROJECT_ID" + +# 2. Grant Cloud SQL Client Access (to run the DB proxy) +gcloud projects add-iam-policy-binding "$PROJECT_ID" \ + --role="roles/cloudsql.client" \ + --member="serviceAccount:$COMPUTE_SA" + +# 3. Grant Cloud SQL Viewer Access (to locate the DB instance connection name) +gcloud projects add-iam-policy-binding "$PROJECT_ID" \ + --role="roles/cloudsql.viewer" \ + --member="serviceAccount:$COMPUTE_SA" + +# 4. Grant GCS Object Admin Access (to upload asset templates) +gcloud projects add-iam-policy-binding "$PROJECT_ID" \ + --role="roles/storage.objectAdmin" \ + --member="serviceAccount:$COMPUTE_SA" +``` + +--- + +### Phase 3: Run Database Seeding inside the VPC +Since your local terminal cannot reach the private IP of the database, you must run the seeding script from a temporary VM inside the VPC. + +#### 1. Setup Temporary Networking & Firewall (in Cloud Shell) +Create a temporary Cloud Router, NAT Gateway (so the VM can pull dependencies from GitHub/npm), and a firewall rule to allow IAP SSH connections: + +```bash +# Create Router +gcloud compute routers create temp-router \ + --network="cs-vpc-development" \ + --region="us-central1" + +# Create NAT Gateway +gcloud compute routers nats create temp-nat \ + --router="temp-router" \ + --region="us-central1" \ + --nat-custom-subnet-ip-ranges="cs-subnet-development" \ + --auto-allocate-nat-external-ips + +# Allow Ingress from Google IAP range to port 22 +gcloud compute firewall-rules create temp-allow-iap-ssh \ + --network="cs-vpc-development" \ + --allow=tcp:22 \ + --source-ranges="35.235.240.0/20" +``` + +#### 2. Create the Temporary VM (in Cloud Shell) +Create a private VM (using `--no-address` to comply with external IP blocks, and Shielded VM configurations to comply with secure boot org policies): + +```bash +gcloud compute instances create temp-seed-vm \ + --zone="us-central1-a" \ + --machine-type="e2-micro" \ + --network="cs-vpc-development" \ + --subnet="cs-subnet-development" \ + --no-address \ + --scopes="https://www.googleapis.com/auth/cloud-platform" \ + --shielded-secure-boot \ + --shielded-vtpm \ + --shielded-integrity-monitoring \ + --metadata="startup-script=sudo apt-get update && sudo apt-get install -y git" +``` + +#### 3. Tunnel SSH and Run the Seeding Script +SSH into the private VM, clone the repository, and run the standalone seeding script: + +```bash +# SSH into the VM (IAP tunnels securely to the private IP) +gcloud compute ssh temp-seed-vm --zone="us-central1-a" --tunnel-through-iap + +# --- Inside the VM Session --- +# Clone the repository +git clone -b private-ip-cloudsql https://github.com/PKAgarwal157/gcc-creative-studio.git ~/gcc-creative-studio + +# Navigate and execute the seeding script +cd ~/gcc-creative-studio +chmod +x seed_only.sh +./seed_only.sh + +# Exit VM when complete +exit +``` + +--- + +### Phase 4: Clean Up & Trigger Deployments + +#### 1. Delete Temporary Resources (in Cloud Shell) +Once seeding is complete, delete the temporary VM and network pathways to avoid costs: + +```bash +# Delete VM +gcloud compute instances delete temp-seed-vm --zone="us-central1-a" --quiet + +# Delete Firewall Rule +gcloud compute firewall-rules delete temp-allow-iap-ssh --quiet + +# Delete NAT Gateway and Router +gcloud compute routers nats delete temp-nat --router="temp-router" --region="us-central1" --quiet +gcloud compute routers delete temp-router --region="us-central1" --quiet +``` + +#### 2. Optional: Manually Trigger Initial Builds (in Cloud Shell) +If you did not trigger the builds during the `bootstrap.sh` script execution (or if you need to redeploy the containers), you can trigger the Cloud Build pipelines manually using these commands: + +```bash +# Trigger Backend +gcloud builds triggers run "cstudio-be-trigger" \ + --branch="private-ip-cloudsql" \ + --project="$PROJECT_ID" \ + --region="us-central1" + +# Trigger Frontend +gcloud builds triggers run "${PROJECT_ID}-trigger" \ + --branch="private-ip-cloudsql" \ + --project="$PROJECT_ID" \ + --region="us-central1" +``` diff --git a/backend/alembic/env.py b/backend/alembic/env.py index 823a8115..db054000 100644 --- a/backend/alembic/env.py +++ b/backend/alembic/env.py @@ -101,7 +101,7 @@ async def alembic_get_connection(): user=config_service.DB_USER, password=config_service.DB_PASS, db=config_service.DB_NAME, - ip_type=IPTypes.PUBLIC, + ip_type=IPTypes.PRIVATE if config_service.DB_IP_TYPE == "PRIVATE" else IPTypes.PUBLIC, ) return conn diff --git a/backend/src/config/config_service.py b/backend/src/config/config_service.py index 8704cdcb..40713fa4 100644 --- a/backend/src/config/config_service.py +++ b/backend/src/config/config_service.py @@ -69,6 +69,7 @@ class ConfigService(BaseSettings): USE_CLOUD_SQL_AUTH_PROXY: bool = False DB_HOST: str = "localhost" DB_PORT: str = "5432" + DB_IP_TYPE: str = "PUBLIC" # --- Veo --- VEO_MODEL_ID: str = "veo-2.0-generate-001" diff --git a/backend/src/database.py b/backend/src/database.py index 7666f348..464ed723 100644 --- a/backend/src/database.py +++ b/backend/src/database.py @@ -131,7 +131,7 @@ async def get_connection(): user=config_service.DB_USER, password=config_service.DB_PASS, db=config_service.DB_NAME, - ip_type=IPTypes.PUBLIC, # Adjust if using Private IP + ip_type=IPTypes.PRIVATE if config_service.DB_IP_TYPE == "PRIVATE" else IPTypes.PUBLIC, ) return conn @@ -196,7 +196,7 @@ async def get_conn(): user=config_service.DB_USER, password=config_service.DB_PASS, db=config_service.DB_NAME, - ip_type=IPTypes.PUBLIC, + ip_type=IPTypes.PRIVATE if config_service.DB_IP_TYPE == "PRIVATE" else IPTypes.PUBLIC, ) self.engine = create_async_engine( diff --git a/bootstrap.sh b/bootstrap.sh index ce088db8..332c5c07 100644 --- a/bootstrap.sh +++ b/bootstrap.sh @@ -157,7 +157,7 @@ start_sql_proxy() { fi # 3. Start Proxy in Background (Port 5432) - ./cloud-sql-proxy --address 0.0.0.0 --port 5432 "$DB_INSTANCE_NAME" > /dev/null 2>&1 & + ./cloud-sql-proxy --address 0.0.0.0 --port 5432 --private-ip "$DB_INSTANCE_NAME" > /dev/null 2>&1 & PROXY_PID=$! export PROXY_PID @@ -449,7 +449,7 @@ configure_environment() { handle_manual_steps() { step 6 "Manual Steps Required"; cd "$REPO_ROOT/infra"; TFVARS_FILE_PATH="$ENV_DIR/$ENV_NAME.tfvars" - info "Enabling required Google Cloud APIs..."; gcloud services enable cloudbuild.googleapis.com secretmanager.googleapis.com firebase.googleapis.com iap.googleapis.com identitytoolkit.googleapis.com texttospeech.googleapis.com workflows.googleapis.com --project="$GCP_PROJECT_ID" + info "Enabling required Google Cloud APIs..."; gcloud services enable cloudbuild.googleapis.com secretmanager.googleapis.com firebase.googleapis.com iap.googleapis.com identitytoolkit.googleapis.com texttospeech.googleapis.com workflows.googleapis.com servicenetworking.googleapis.com --project="$GCP_PROJECT_ID" if [ -z "$GITHUB_CONN_NAME" ]; then prompt "\nDo you already have a Cloud Build Host Connection for GitHub in this project? (y/n)"; read -r REPLY < /dev/tty if [[ $REPLY =~ ^[Yy]$ ]]; then prompt "Please enter the existing connection name:"; read -p " Connection Name: " GITHUB_CONN_NAME < /dev/tty @@ -793,7 +793,6 @@ main() { "populate_oauth_secrets" "update_oauth_client" "update_secrets" - "seed_data" "trigger_builds" ) for i in "${!steps_to_run[@]}"; do diff --git a/infra/environments/dev-infra-example/dev.tfvars b/infra/environments/dev-infra-example/dev.tfvars index 547b509a..da81ce0d 100644 --- a/infra/environments/dev-infra-example/dev.tfvars +++ b/infra/environments/dev-infra-example/dev.tfvars @@ -21,6 +21,7 @@ frontend_custom_audiences = ["YOUR_OAUTH_WEB_CLIENT_ID_HERE", "YOUR_GCP_PROJECT_ be_env_vars = { common = { LOG_LEVEL = "INFO" + DB_IP_TYPE = "PRIVATE" } development = { ENVIRONMENT = "development" diff --git a/infra/modules/cloud-run-service/main.tf b/infra/modules/cloud-run-service/main.tf index a3c96a06..cf928dc8 100644 --- a/infra/modules/cloud-run-service/main.tf +++ b/infra/modules/cloud-run-service/main.tf @@ -39,6 +39,14 @@ resource "google_cloud_run_v2_service" "this" { template { service_account = google_service_account.run_sa.email + + vpc_access { + network_interfaces { + network = var.vpc_network_id + subnetwork = var.vpc_subnetwork_id + } + egress = "ALL_TRAFFIC" + } volumes { name = "cloudsql" cloud_sql_instance { diff --git a/infra/modules/cloud-run-service/variables.tf b/infra/modules/cloud-run-service/variables.tf index 2ae93530..79675898 100644 --- a/infra/modules/cloud-run-service/variables.tf +++ b/infra/modules/cloud-run-service/variables.tf @@ -129,3 +129,14 @@ variable "db_secret_id" { } variable "db_name" { type = string } variable "db_user" { type = string } + +variable "vpc_network_id" { + description = "The ID of the VPC network" + type = string +} + +variable "vpc_subnetwork_id" { + description = "The ID of the subnetwork" + type = string +} + diff --git a/infra/modules/network/main.tf b/infra/modules/network/main.tf new file mode 100644 index 00000000..c909657e --- /dev/null +++ b/infra/modules/network/main.tf @@ -0,0 +1,49 @@ +# Copyright 2025 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +resource "google_compute_network" "vpc" { + name = "cs-vpc-${var.environment}" + auto_create_subnetworks = false + project = var.project_id +} + +resource "google_compute_subnetwork" "subnet" { + name = "cs-subnet-${var.environment}" + ip_cidr_range = "10.0.0.0/24" + region = var.region + network = google_compute_network.vpc.id + project = var.project_id + private_ip_google_access = true + + log_config { + aggregation_interval = "INTERVAL_5_SEC" + flow_sampling = 0.5 + metadata = "INCLUDE_ALL_METADATA" + } +} + +resource "google_compute_global_address" "private_ip_address" { + name = "cs-private-ip-address" + purpose = "VPC_PEERING" + address_type = "INTERNAL" + prefix_length = 16 + network = google_compute_network.vpc.id + project = var.project_id +} + +resource "google_service_networking_connection" "private_vpc_connection" { + network = google_compute_network.vpc.id + service = "servicenetworking.googleapis.com" + reserved_peering_ranges = [google_compute_global_address.private_ip_address.name] +} diff --git a/infra/modules/network/outputs.tf b/infra/modules/network/outputs.tf new file mode 100644 index 00000000..f4c319b3 --- /dev/null +++ b/infra/modules/network/outputs.tf @@ -0,0 +1,33 @@ +# Copyright 2025 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +output "network_id" { + description = "The ID of the VPC network" + value = google_compute_network.vpc.id +} + +output "network_name" { + description = "The name of the VPC network" + value = google_compute_network.vpc.name +} + +output "subnetwork_id" { + description = "The ID of the subnetwork" + value = google_compute_subnetwork.subnet.id +} + +output "private_vpc_connection_id" { + description = "The ID of the private VPC connection" + value = google_service_networking_connection.private_vpc_connection.id +} diff --git a/infra/modules/network/variables.tf b/infra/modules/network/variables.tf new file mode 100644 index 00000000..423ffacb --- /dev/null +++ b/infra/modules/network/variables.tf @@ -0,0 +1,28 @@ +# Copyright 2025 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +variable "project_id" { + type = string + description = "The GCP Project ID." +} + +variable "region" { + type = string + description = "The region to deploy GKE and network resources." +} + +variable "environment" { + type = string + description = "The deployment environment (e.g., dev, prod)." +} diff --git a/infra/modules/platform/main.tf b/infra/modules/platform/main.tf index 1c542c4e..6638f07e 100644 --- a/infra/modules/platform/main.tf +++ b/infra/modules/platform/main.tf @@ -86,6 +86,13 @@ data "google_secret_manager_secret_version" "db_password" { version = "latest" } +module "network" { + source = "../network" + project_id = var.gcp_project_id + region = var.gcp_region + environment = var.environment +} + # 2. Call PostgreSQL Module module "postgresql" { source = "../postgresql" @@ -94,6 +101,10 @@ module "postgresql" { # Pass the ACTUAL value to create the user db_password = data.google_secret_manager_secret_version.db_password.secret_data + + vpc_network_id = module.network.network_id + + depends_on = [module.network] } # --- Service Module Calls --- @@ -132,6 +143,9 @@ module "backend_service" { # Pass the Secret ID reference (NOT the value) for Cloud Run db_secret_id = "creative-studio-db-password" + + vpc_network_id = module.network.network_id + vpc_subnetwork_id = module.network.subnetwork_id } resource "google_firebase_project" "default" { diff --git a/infra/modules/postgresql/main.tf b/infra/modules/postgresql/main.tf index 22398997..945d0161 100644 --- a/infra/modules/postgresql/main.tf +++ b/infra/modules/postgresql/main.tf @@ -32,7 +32,8 @@ resource "google_sql_database_instance" "default" { } ip_configuration { - ipv4_enabled = true # Easy connectivity from Cloud Run without VPC peering complexity + ipv4_enabled = false + private_network = var.vpc_network_id } } diff --git a/infra/modules/postgresql/variables.tf b/infra/modules/postgresql/variables.tf index 63c7bdd1..8e97831f 100644 --- a/infra/modules/postgresql/variables.tf +++ b/infra/modules/postgresql/variables.tf @@ -17,3 +17,9 @@ variable "region" {} variable "db_name" { default = "creative_studio" } variable "db_user" { default = "studio_user" } variable "db_password" { sensitive = true } + +variable "vpc_network_id" { + type = string + description = "The ID of the VPC network where the private IP will be allocated." +} + diff --git a/seed_only.sh b/seed_only.sh new file mode 100644 index 00000000..4b264905 --- /dev/null +++ b/seed_only.sh @@ -0,0 +1,106 @@ +#!/bin/bash +# A standalone script to run database seeding directly from the private jump box VM. +# It bypasses Terraform, Node.js, and Firebase requirements, needing only gcloud and python. + +set -e + +# Helper colors +C_RESET='\033[0m' +C_GREEN='\033[1;32m' +C_RED='\033[1;31m' +C_CYAN='\033[1;36m' +C_YELLOW='\033[1;33m' + +info() { echo -e "${C_CYAN}➡️ $1${C_RESET}"; } +success() { echo -e "${C_GREEN}✅ $1${C_RESET}"; } +fail() { echo -e "${C_RED}❌ $1${C_RESET}" >&2; exit 1; } +warn() { echo -e "${C_YELLOW}⚠️ $1${C_RESET}"; } + +# Detect Project ID +GCP_PROJECT_ID=$(gcloud config get-value project 2>/dev/null) +if [ -z "$GCP_PROJECT_ID" ]; then + fail "Could not determine active gcloud project. Please run 'gcloud config set project [ID]' first." +fi + +info "Using Project: ${C_YELLOW}${GCP_PROJECT_ID}${C_RESET}" + +# 1. Resolve Cloud SQL connection name using gcloud +info "Locating Cloud SQL private instance..." +DB_INSTANCE_NAME=$(gcloud sql instances list --format="value(connectionName)" --filter="name:creative-studio-db*" --project="$GCP_PROJECT_ID" | head -n 1) + +if [ -z "$DB_INSTANCE_NAME" ]; then + fail "Could not find active Cloud SQL instance in project $GCP_PROJECT_ID." +fi +info "Found database instance: ${C_YELLOW}${DB_INSTANCE_NAME}${C_RESET}" + +# 2. Fetch password from Secret Manager +info "Retrieving database password..." +DB_PASS=$(gcloud secrets versions access latest --secret="creative-studio-db-password" --project="$GCP_PROJECT_ID" || fail "Failed to read secret 'creative-studio-db-password' from Secret Manager.") + +export DB_USER="studio_user" +export DB_PASS="$DB_PASS" +export DB_NAME="creative_studio" +export DB_HOST="127.0.0.1" +export DB_PORT="5432" +export USE_CLOUD_SQL_AUTH_PROXY=true + +# 3. Start Cloud SQL Auth Proxy +if [ ! -f "cloud-sql-proxy" ]; then + info "Downloading Cloud SQL Auth Proxy..." + curl -o cloud-sql-proxy https://storage.googleapis.com/cloud-sql-connectors/cloud-sql-proxy/v2.8.0/cloud-sql-proxy.linux.amd64 + chmod +x cloud-sql-proxy +fi + +info "Starting Cloud SQL Proxy (Private IP)..." +./cloud-sql-proxy --address 0.0.0.0 --port 5432 --private-ip "$DB_INSTANCE_NAME" > /dev/null 2>&1 & +PROXY_PID=$! +export PROXY_PID + +# Ensure proxy stops on exit +stop_proxy() { + if [ -n "$PROXY_PID" ]; then + info "Stopping Cloud SQL Proxy..." + kill "$PROXY_PID" 2>/dev/null || true + fi +} +trap stop_proxy EXIT + +# Wait for proxy readiness +echo -n " Waiting for proxy connection..." +for i in {1..30}; do + if (echo > /dev/tcp/127.0.0.1/5432) >/dev/null 2>&1; then + echo " Connected!" + break + fi + echo -n "." + sleep 1 +done + +# 4. Setup Environment and run Python Seeding Script +CURRENT_USER=$(gcloud config get-value account 2>/dev/null || echo "system") +ASSET_BUCKET_NAME="${GCP_PROJECT_ID}-cs-development-bucket" + +export GOOGLE_CLOUD_PROJECT=$GCP_PROJECT_ID +export ADMIN_USER_EMAIL=$CURRENT_USER +export GENMEDIA_BUCKET=$ASSET_BUCKET_NAME + +# Install uv if missing +if ! command -v uv >/dev/null; then + info "Installing uv..." + curl -LsSf https://astral.sh/uv/install.sh | sh + export PATH="$HOME/.local/bin:$PATH" +fi + +info "Setting up Python virtual environment..." +VENV_DIR="$(pwd)/backend/.venv" +uv venv "$VENV_DIR" --python 3.12 --clear + +info "Installing dependencies from pyproject.toml..." +uv pip install --python "$VENV_DIR/bin/python" -e backend + +info "Running seeding script..." +if (cd backend && "$VENV_DIR/bin/python" -m bootstrap.bootstrap); then + success "Database seeded successfully." +else + fail "Database seeding failed." +fi