278 lines
8.4 KiB
Markdown
278 lines
8.4 KiB
Markdown
# Google Secret Manager Integration Guide
|
|
|
|
This guide explains how to integrate Google Cloud Secret Manager with your Node.js application to securely manage environment variables.
|
|
|
|
## Overview
|
|
|
|
The Google Secret Manager integration allows you to:
|
|
- Store sensitive configuration values (passwords, API keys, tokens) in Google Cloud Secret Manager
|
|
- Load secrets at application startup and merge them with your existing environment variables
|
|
- Maintain backward compatibility with `.env` files for local development
|
|
- Use minimal code changes - existing `process.env.VARIABLE_NAME` access continues to work
|
|
|
|
## Prerequisites
|
|
|
|
1. **Google Cloud Project** with Secret Manager API enabled
|
|
2. **Service Account** with Secret Manager Secret Accessor role
|
|
3. **Authentication** - Service account credentials configured (via `GCP_KEY_FILE` or default credentials)
|
|
|
|
## Setup Instructions
|
|
|
|
### 1. Enable Secret Manager API
|
|
|
|
```bash
|
|
gcloud services enable secretmanager.googleapis.com --project=YOUR_PROJECT_ID
|
|
```
|
|
|
|
### 2. Create Secrets in Google Secret Manager
|
|
|
|
Create secrets using the Google Cloud Console or gcloud CLI:
|
|
|
|
```bash
|
|
# Example: Create a database password secret
|
|
echo -n "your-secure-password" | gcloud secrets create DB_PASSWORD \
|
|
--project=YOUR_PROJECT_ID \
|
|
--data-file=-
|
|
|
|
# Example: Create a JWT secret
|
|
echo -n "your-jwt-secret-key" | gcloud secrets create JWT_SECRET \
|
|
--project=YOUR_PROJECT_ID \
|
|
--data-file=-
|
|
|
|
# Grant service account access to secrets
|
|
gcloud secrets add-iam-policy-binding DB_PASSWORD \
|
|
--member="serviceAccount:YOUR_SERVICE_ACCOUNT@YOUR_PROJECT.iam.gserviceaccount.com" \
|
|
--role="roles/secretmanager.secretAccessor" \
|
|
--project=YOUR_PROJECT_ID
|
|
```
|
|
|
|
### 3. Configure Environment Variables
|
|
|
|
Add the following to your `.env` file:
|
|
|
|
```env
|
|
# Google Secret Manager Configuration
|
|
USE_GOOGLE_SECRET_MANAGER=true
|
|
GCP_PROJECT_ID=your-project-id
|
|
|
|
# Optional: Prefix for all secret names (e.g., "prod" -> looks for "prod-DB_PASSWORD")
|
|
GCP_SECRET_PREFIX=
|
|
|
|
# Optional: JSON file mapping secret names to env var names
|
|
GCP_SECRET_MAP_FILE=./secret-map.json
|
|
```
|
|
|
|
**Important Notes:**
|
|
- Set `USE_GOOGLE_SECRET_MANAGER=true` to enable the integration
|
|
- `GCP_PROJECT_ID` must be set (same as used for GCS/Vertex AI)
|
|
- `GCP_KEY_FILE` should already be configured for other GCP services
|
|
- When `USE_GOOGLE_SECRET_MANAGER=false` or not set, the app uses `.env` file only
|
|
|
|
### 4. Secret Name Mapping
|
|
|
|
By default, secrets in Google Secret Manager are automatically mapped to environment variables:
|
|
- Secret name: `DB_PASSWORD` → Environment variable: `DB_PASSWORD`
|
|
- Secret name: `db-password` → Environment variable: `DB_PASSWORD` (hyphens converted to underscores, uppercase)
|
|
- Secret name: `jwt-secret-key` → Environment variable: `JWT_SECRET_KEY`
|
|
|
|
#### Custom Mapping (Optional)
|
|
|
|
If you need custom mappings, create a JSON file (e.g., `secret-map.json`):
|
|
|
|
```json
|
|
{
|
|
"db-password-prod": "DB_PASSWORD",
|
|
"jwt-secret-key": "JWT_SECRET",
|
|
"okta-client-secret-prod": "OKTA_CLIENT_SECRET"
|
|
}
|
|
```
|
|
|
|
Then set in `.env`:
|
|
```env
|
|
GCP_SECRET_MAP_FILE=./secret-map.json
|
|
```
|
|
|
|
### 5. Secret Prefix (Optional)
|
|
|
|
If all your secrets share a common prefix:
|
|
|
|
```env
|
|
GCP_SECRET_PREFIX=prod
|
|
```
|
|
|
|
This will look for secrets named `prod-DB_PASSWORD`, `prod-JWT_SECRET`, etc.
|
|
|
|
## How It Works
|
|
|
|
1. **Application Startup:**
|
|
- `.env` file is loaded first (provides fallback values)
|
|
- If `USE_GOOGLE_SECRET_MANAGER=true`, secrets are fetched from Google Secret Manager
|
|
- Secrets are merged into `process.env`, overriding `.env` values if they exist
|
|
- Application continues with merged environment variables
|
|
|
|
2. **Fallback Behavior:**
|
|
- If Secret Manager is disabled or fails, the app falls back to `.env` file
|
|
- No errors are thrown - the app continues with available configuration
|
|
- Logs indicate whether secrets were loaded successfully
|
|
|
|
3. **Existing Code Compatibility:**
|
|
- No changes needed to existing code
|
|
- Continue using `process.env.VARIABLE_NAME` as before
|
|
- Secrets from GCS automatically populate `process.env`
|
|
|
|
## Default Secrets Loaded
|
|
|
|
The service automatically attempts to load these common secrets (if they exist in Secret Manager):
|
|
|
|
**Database:**
|
|
- `DB_HOST`, `DB_PORT`, `DB_NAME`, `DB_USER`, `DB_PASSWORD`
|
|
|
|
**Authentication:**
|
|
- `JWT_SECRET`, `REFRESH_TOKEN_SECRET`, `SESSION_SECRET`
|
|
|
|
**SSO/Okta:**
|
|
- `OKTA_DOMAIN`, `OKTA_CLIENT_ID`, `OKTA_CLIENT_SECRET`, `OKTA_API_TOKEN`
|
|
|
|
**Email:**
|
|
- `SMTP_HOST`, `SMTP_PORT`, `SMTP_USER`, `SMTP_PASSWORD`
|
|
|
|
**Web Push (VAPID):**
|
|
- `VAPID_PUBLIC_KEY`, `VAPID_PRIVATE_KEY`
|
|
|
|
**Logging:**
|
|
- `LOKI_HOST`, `LOKI_USER`, `LOKI_PASSWORD`
|
|
|
|
### Loading Custom Secrets
|
|
|
|
To load additional secrets, modify the code:
|
|
|
|
```typescript
|
|
// In server.ts or app.ts
|
|
import { googleSecretManager } from './services/googleSecretManager.service';
|
|
|
|
// Load default secrets + custom ones
|
|
await googleSecretManager.loadSecrets([
|
|
'DB_PASSWORD',
|
|
'JWT_SECRET',
|
|
'CUSTOM_API_KEY', // Your custom secret
|
|
'CUSTOM_SECRET_2'
|
|
]);
|
|
```
|
|
|
|
Or load a single secret on-demand:
|
|
|
|
```typescript
|
|
import { googleSecretManager } from './services/googleSecretManager.service';
|
|
|
|
const apiKey = await googleSecretManager.getSecretValue('CUSTOM_API_KEY', 'API_KEY');
|
|
```
|
|
|
|
## Security Best Practices
|
|
|
|
1. **Service Account Permissions:**
|
|
- Grant only `roles/secretmanager.secretAccessor` role
|
|
- Use separate service accounts for different environments
|
|
- Never grant `roles/owner` or `roles/editor` to service accounts
|
|
|
|
2. **Secret Rotation:**
|
|
- Rotate secrets regularly in Google Secret Manager
|
|
- The app automatically uses the `latest` version of each secret
|
|
- No code changes needed when secrets are rotated
|
|
|
|
3. **Environment Separation:**
|
|
- Use different Google Cloud projects for dev/staging/prod
|
|
- Use `GCP_SECRET_PREFIX` to namespace secrets by environment
|
|
- Never commit `.env` files with production secrets to version control
|
|
|
|
4. **Access Control:**
|
|
- Use IAM policies to control who can read secrets
|
|
- Enable audit logging for secret access
|
|
- Regularly review secret access logs
|
|
|
|
## Troubleshooting
|
|
|
|
### Secrets Not Loading
|
|
|
|
**Check logs for:**
|
|
```
|
|
[Secret Manager] Google Secret Manager is disabled (USE_GOOGLE_SECRET_MANAGER != true)
|
|
[Secret Manager] GCP_PROJECT_ID not set, skipping Google Secret Manager
|
|
[Secret Manager] Failed to load secrets: [error message]
|
|
```
|
|
|
|
**Common issues:**
|
|
1. `USE_GOOGLE_SECRET_MANAGER` not set to `true`
|
|
2. `GCP_PROJECT_ID` not configured
|
|
3. Service account lacks Secret Manager permissions
|
|
4. Secrets don't exist in Secret Manager
|
|
5. Incorrect secret names (check case sensitivity)
|
|
|
|
### Service Account Authentication
|
|
|
|
Ensure service account credentials are available:
|
|
- Set `GCP_KEY_FILE` to point to service account JSON file
|
|
- Or configure Application Default Credentials (ADC)
|
|
- Test with: `gcloud auth application-default login`
|
|
|
|
### Secret Not Found
|
|
|
|
If a secret doesn't exist in Secret Manager:
|
|
- The app logs a debug message and continues
|
|
- Falls back to `.env` file value
|
|
- This is expected behavior - not all secrets need to be in GCS
|
|
|
|
### Debugging
|
|
|
|
Enable debug logging by setting:
|
|
```env
|
|
LOG_LEVEL=debug
|
|
```
|
|
|
|
This will show detailed logs about which secrets are being loaded.
|
|
|
|
## Example Configuration
|
|
|
|
**Local Development (.env):**
|
|
```env
|
|
USE_GOOGLE_SECRET_MANAGER=false
|
|
DB_PASSWORD=local-dev-password
|
|
JWT_SECRET=local-jwt-secret
|
|
```
|
|
|
|
**Production (.env):**
|
|
```env
|
|
USE_GOOGLE_SECRET_MANAGER=true
|
|
GCP_PROJECT_ID=re-platform-workflow-dealer
|
|
GCP_SECRET_PREFIX=prod
|
|
GCP_KEY_FILE=./credentials/service-account.json
|
|
# DB_PASSWORD and other secrets loaded from GCS
|
|
```
|
|
|
|
## Migration Strategy
|
|
|
|
1. **Phase 1: Setup**
|
|
- Create secrets in Google Secret Manager
|
|
- Keep `.env` file with current values (as backup)
|
|
|
|
2. **Phase 2: Test**
|
|
- Set `USE_GOOGLE_SECRET_MANAGER=true` in development
|
|
- Verify secrets are loaded correctly
|
|
- Test application functionality
|
|
|
|
3. **Phase 3: Production**
|
|
- Deploy with `USE_GOOGLE_SECRET_MANAGER=true`
|
|
- Monitor logs for secret loading success
|
|
- Remove sensitive values from `.env` file (keep placeholders)
|
|
|
|
4. **Phase 4: Cleanup**
|
|
- Remove production secrets from `.env` file
|
|
- Ensure all secrets are in Secret Manager
|
|
- Document secret names and mappings
|
|
|
|
## Additional Resources
|
|
|
|
- [Google Secret Manager Documentation](https://cloud.google.com/secret-manager/docs)
|
|
- [Secret Manager Client Library](https://cloud.google.com/nodejs/docs/reference/secret-manager/latest)
|
|
- [Service Account Best Practices](https://cloud.google.com/iam/docs/best-practices-service-accounts)
|
|
|