laxmanhalaki/Re_Backend
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
87273ae044
Re_Backend/docs/GOOGLE_SECRET_MANAGER_SETUP.md

8.4 KiB
Raw Blame History

Google Secret Manager Integration Guide

This guide explains how to integrate Google Cloud Secret Manager with your Node.js application to securely manage environment variables.

Overview

The Google Secret Manager integration allows you to:

  • Store sensitive configuration values (passwords, API keys, tokens) in Google Cloud Secret Manager
  • Load secrets at application startup and merge them with your existing environment variables
  • Maintain backward compatibility with .env files for local development
  • Use minimal code changes - existing process.env.VARIABLE_NAME access continues to work

Prerequisites

  1. Google Cloud Project with Secret Manager API enabled
  2. Service Account with Secret Manager Secret Accessor role
  3. Authentication - Service account credentials configured (via GCP_KEY_FILE or default credentials)

Setup Instructions

1. Enable Secret Manager API

gcloud services enable secretmanager.googleapis.com --project=YOUR_PROJECT_ID

2. Create Secrets in Google Secret Manager

Create secrets using the Google Cloud Console or gcloud CLI:

# Example: Create a database password secret
echo -n "your-secure-password" | gcloud secrets create DB_PASSWORD \
  --project=YOUR_PROJECT_ID \
  --data-file=-

# Example: Create a JWT secret
echo -n "your-jwt-secret-key" | gcloud secrets create JWT_SECRET \
  --project=YOUR_PROJECT_ID \
  --data-file=-

# Grant service account access to secrets
gcloud secrets add-iam-policy-binding DB_PASSWORD \
  --member="serviceAccount:YOUR_SERVICE_ACCOUNT@YOUR_PROJECT.iam.gserviceaccount.com" \
  --role="roles/secretmanager.secretAccessor" \
  --project=YOUR_PROJECT_ID

3. Configure Environment Variables

Add the following to your .env file:

# Google Secret Manager Configuration
USE_GOOGLE_SECRET_MANAGER=true
GCP_PROJECT_ID=your-project-id

# Optional: Prefix for all secret names (e.g., "prod" -> looks for "prod-DB_PASSWORD")
GCP_SECRET_PREFIX=

# Optional: JSON file mapping secret names to env var names
GCP_SECRET_MAP_FILE=./secret-map.json

Important Notes:

  • Set USE_GOOGLE_SECRET_MANAGER=true to enable the integration
  • GCP_PROJECT_ID must be set (same as used for GCS/Vertex AI)
  • GCP_KEY_FILE should already be configured for other GCP services
  • When USE_GOOGLE_SECRET_MANAGER=false or not set, the app uses .env file only

4. Secret Name Mapping

By default, secrets in Google Secret Manager are automatically mapped to environment variables:

  • Secret name: DB_PASSWORD → Environment variable: DB_PASSWORD
  • Secret name: db-password → Environment variable: DB_PASSWORD (hyphens converted to underscores, uppercase)
  • Secret name: jwt-secret-key → Environment variable: JWT_SECRET_KEY

Custom Mapping (Optional)

If you need custom mappings, create a JSON file (e.g., secret-map.json):

{
  "db-password-prod": "DB_PASSWORD",
  "jwt-secret-key": "JWT_SECRET",
  "okta-client-secret-prod": "OKTA_CLIENT_SECRET"
}

Then set in .env:

GCP_SECRET_MAP_FILE=./secret-map.json

5. Secret Prefix (Optional)

If all your secrets share a common prefix:

GCP_SECRET_PREFIX=prod

This will look for secrets named prod-DB_PASSWORD, prod-JWT_SECRET, etc.

How It Works

  1. Application Startup:

    • .env file is loaded first (provides fallback values)
    • If USE_GOOGLE_SECRET_MANAGER=true, secrets are fetched from Google Secret Manager
    • Secrets are merged into process.env, overriding .env values if they exist
    • Application continues with merged environment variables

  2. Fallback Behavior:

    • If Secret Manager is disabled or fails, the app falls back to .env file
    • No errors are thrown - the app continues with available configuration
    • Logs indicate whether secrets were loaded successfully

  3. Existing Code Compatibility:

    • No changes needed to existing code
    • Continue using process.env.VARIABLE_NAME as before
    • Secrets from GCS automatically populate process.env

Default Secrets Loaded

The service automatically attempts to load these common secrets (if they exist in Secret Manager):

Database:

  • DB_HOST, DB_PORT, DB_NAME, DB_USER, DB_PASSWORD

Authentication:

  • JWT_SECRET, REFRESH_TOKEN_SECRET, SESSION_SECRET

SSO/Okta:

  • OKTA_DOMAIN, OKTA_CLIENT_ID, OKTA_CLIENT_SECRET, OKTA_API_TOKEN

Email:

  • SMTP_HOST, SMTP_PORT, SMTP_USER, SMTP_PASSWORD

Web Push (VAPID):

  • VAPID_PUBLIC_KEY, VAPID_PRIVATE_KEY

Logging:

  • LOKI_HOST, LOKI_USER, LOKI_PASSWORD

Loading Custom Secrets

To load additional secrets, modify the code:

// In server.ts or app.ts
import { googleSecretManager } from './services/googleSecretManager.service';

// Load default secrets + custom ones
await googleSecretManager.loadSecrets([
  'DB_PASSWORD',
  'JWT_SECRET',
  'CUSTOM_API_KEY',  // Your custom secret
  'CUSTOM_SECRET_2'
]);

Or load a single secret on-demand:

import { googleSecretManager } from './services/googleSecretManager.service';

const apiKey = await googleSecretManager.getSecretValue('CUSTOM_API_KEY', 'API_KEY');

Security Best Practices

  1. Service Account Permissions:

    • Grant only roles/secretmanager.secretAccessor role
    • Use separate service accounts for different environments
    • Never grant roles/owner or roles/editor to service accounts

  2. Secret Rotation:

    • Rotate secrets regularly in Google Secret Manager
    • The app automatically uses the latest version of each secret
    • No code changes needed when secrets are rotated

  3. Environment Separation:

    • Use different Google Cloud projects for dev/staging/prod
    • Use GCP_SECRET_PREFIX to namespace secrets by environment
    • Never commit .env files with production secrets to version control

  4. Access Control:

    • Use IAM policies to control who can read secrets
    • Enable audit logging for secret access
    • Regularly review secret access logs

Troubleshooting

Secrets Not Loading

Check logs for:

[Secret Manager] Google Secret Manager is disabled (USE_GOOGLE_SECRET_MANAGER != true)
[Secret Manager] GCP_PROJECT_ID not set, skipping Google Secret Manager
[Secret Manager] Failed to load secrets: [error message]

Common issues:

  1. USE_GOOGLE_SECRET_MANAGER not set to true
  2. GCP_PROJECT_ID not configured
  3. Service account lacks Secret Manager permissions
  4. Secrets don't exist in Secret Manager
  5. Incorrect secret names (check case sensitivity)

Service Account Authentication

Ensure service account credentials are available:

  • Set GCP_KEY_FILE to point to service account JSON file
  • Or configure Application Default Credentials (ADC)
  • Test with: gcloud auth application-default login

Secret Not Found

If a secret doesn't exist in Secret Manager:

  • The app logs a debug message and continues
  • Falls back to .env file value
  • This is expected behavior - not all secrets need to be in GCS

Debugging

Enable debug logging by setting:

LOG_LEVEL=debug

This will show detailed logs about which secrets are being loaded.

Example Configuration

Local Development (.env):

USE_GOOGLE_SECRET_MANAGER=false
DB_PASSWORD=local-dev-password
JWT_SECRET=local-jwt-secret

Production (.env):

USE_GOOGLE_SECRET_MANAGER=true
GCP_PROJECT_ID=re-platform-workflow-dealer
GCP_SECRET_PREFIX=prod
GCP_KEY_FILE=./credentials/service-account.json
# DB_PASSWORD and other secrets loaded from GCS

Migration Strategy

  1. Phase 1: Setup

    • Create secrets in Google Secret Manager
    • Keep .env file with current values (as backup)

  2. Phase 2: Test

    • Set USE_GOOGLE_SECRET_MANAGER=true in development
    • Verify secrets are loaded correctly
    • Test application functionality

  3. Phase 3: Production

    • Deploy with USE_GOOGLE_SECRET_MANAGER=true
    • Monitor logs for secret loading success
    • Remove sensitive values from .env file (keep placeholders)

  4. Phase 4: Cleanup

    • Remove production secrets from .env file
    • Ensure all secrets are in Secret Manager
    • Document secret names and mappings

Additional Resources