backend changes

REQUIREMENT_PROCESSOR_MIGRATION_FIX.md

@@ -0,0 +1,121 @@
+# Deployment Fix Guide - Requirement Processor Migration Issue
+
+## Problem Summary
+The deployment failed due to a database migration constraint issue in the requirement processor service. The error was:
+```
+❌ Migration failed: 001_business_context_tables.sql - null value in column "service" of relation "schema_migrations" violates not-null constraint
+```
+
+## Root Cause
+The requirement processor's migration system was using an outdated schema for the `schema_migrations` table that didn't include the required `service` field, while the main database migration system expected this field to be present and non-null.
+
+## Fix Applied
+
+### 1. Updated Migration Script (`migrate.py`)
+- ✅ Updated `schema_migrations` table schema to include `service` field
+- ✅ Modified `is_applied()` function to check by both version and service
+- ✅ Updated `mark_applied()` function to include service and description
+- ✅ Fixed `run_migration()` function to use service parameter
+
+### 2. Fixed Migration Files
+- ✅ Removed foreign key constraint from initial migration to avoid dependency issues
+- ✅ The second migration already handles the constraint properly
+
+### 3. Created Fix Script
+- ✅ Created `scripts/fix-requirement-processor-migration.sh` to clean up and restart the service
+
+## Deployment Steps
+
+### Option 1: Use the Fix Script (Recommended)
+```bash
+cd /home/ubuntu/codenuk-backend-live
+./scripts/fix-requirement-processor-migration.sh
+```
+
+### Option 2: Manual Fix
+```bash
+# 1. Stop the requirement processor
+docker compose stop requirement-processor
+
+# 2. Clean up failed migration records
+PGPASSWORD="password" psql -h localhost -p 5432 -U postgres -d dev_pipeline << 'EOF'
+DELETE FROM schema_migrations WHERE service = 'requirement-processor' OR version LIKE '%.sql';
+EOF
+
+# 3. Restart the service
+docker compose up -d requirement-processor
+
+# 4. Check status
+docker compose ps requirement-processor
+```
+
+### Option 3: Full Redeploy
+```bash
+# Stop all services
+docker compose down
+
+# Clean up database (if needed)
+PGPASSWORD="password" psql -h localhost -p 5432 -U postgres -d dev_pipeline << 'EOF'
+DELETE FROM schema_migrations WHERE service = 'requirement-processor';
+EOF
+
+# Start all services
+docker compose up -d
+```
+
+## Verification Steps
+
+1. **Check Service Status**
+   ```bash
+   docker compose ps requirement-processor
+   ```
+
+2. **Check Migration Records**
+   ```bash
+   PGPASSWORD="password" psql -h localhost -p 5432 -U postgres -d dev_pipeline << 'EOF'
+   SELECT service, version, applied_at, description 
+   FROM schema_migrations 
+   WHERE service = 'requirement-processor' 
+   ORDER BY applied_at;
+   EOF
+   ```
+
+3. **Check Service Logs**
+   ```bash
+   docker compose logs requirement-processor
+   ```
+
+4. **Test Health Endpoint**
+   ```bash
+   curl http://localhost:8001/health
+   ```
+
+## Expected Results
+
+After the fix:
+- ✅ Requirement processor service should start successfully
+- ✅ Migration records should show proper service field
+- ✅ Health endpoint should return 200 OK
+- ✅ All other services should continue running normally
+
+## Prevention
+
+To prevent this issue in the future:
+1. Always ensure migration scripts use the correct `schema_migrations` table schema
+2. Include service field in all migration tracking
+3. Test migrations in development before deploying to production
+4. Use the shared migration system consistently across all services
+
+## Troubleshooting
+
+If the issue persists:
+1. Check database connectivity
+2. Verify PostgreSQL is running
+3. Check disk space and memory
+4. Review all service logs
+5. Consider a full database reset if necessary
+
+## Files Modified
+- `services/requirement-processor/migrations/migrate.py` - Updated migration system
+- `services/requirement-processor/migrations/001_business_context_tables.sql` - Removed FK constraint
+- `scripts/fix-requirement-processor-migration.sh` - Created fix script

scripts/fix-requirement-processor-migration.sh

@@ -0,0 +1,80 @@
+#!/bin/bash
+
+# Fix Requirement Processor Migration Issue
+# This script fixes the schema_migrations constraint issue
+
+set -e
+
+# Colors for output
+RED='\033[0;31m'
+GREEN='\033[0;32m'
+YELLOW='\033[1;33m'
+NC='\033[0m' # No Color
+
+log() {
+    echo -e "${GREEN}[$(date +'%Y-%m-%d %H:%M:%S')]${NC} $1"
+}
+
+warn() {
+    echo -e "${YELLOW}[$(date +'%Y-%m-%d %H:%M:%S')] WARNING:${NC} $1"
+}
+
+error() {
+    echo -e "${RED}[$(date +'%Y-%m-%d %H:%M:%S')] ERROR:${NC} $1"
+}
+
+# Database connection settings
+DB_HOST=${DB_HOST:-"localhost"}
+DB_PORT=${DB_PORT:-"5432"}
+DB_USER=${DB_USER:-"postgres"}
+DB_NAME=${DB_NAME:-"dev_pipeline"}
+DB_PASSWORD=${DB_PASSWORD:-"password"}
+
+log "🔧 Fixing Requirement Processor Migration Issue"
+log "=============================================="
+
+# Check if we're in the right directory
+if [ ! -f "docker-compose.yml" ]; then
+    error "Please run this script from the codenuk-backend-live directory"
+    exit 1
+fi
+
+log "📋 Step 1: Stopping the requirement-processor service"
+docker compose stop requirement-processor || true
+
+log "📋 Step 2: Cleaning up failed migration records"
+PGPASSWORD="$DB_PASSWORD" psql -h "$DB_HOST" -p "$DB_PORT" -U "$DB_USER" -d "$DB_NAME" << 'EOF'
+-- Remove any failed migration records for requirement-processor
+DELETE FROM schema_migrations WHERE service = 'requirement-processor' OR version LIKE '%.sql';
+
+-- Ensure the schema_migrations table has the correct structure
+ALTER TABLE schema_migrations ALTER COLUMN service SET NOT NULL;
+EOF
+
+log "📋 Step 3: Restarting the requirement-processor service"
+docker compose up -d requirement-processor
+
+log "📋 Step 4: Waiting for service to be healthy"
+sleep 10
+
+# Check if the service is running
+if docker compose ps requirement-processor | grep -q "Up"; then
+    log "✅ Requirement processor service is running"
+else
+    error "❌ Requirement processor service failed to start"
+    docker compose logs requirement-processor
+    exit 1
+fi
+
+log "📋 Step 5: Verifying migration status"
+PGPASSWORD="$DB_PASSWORD" psql -h "$DB_HOST" -p "$DB_PORT" -U "$DB_USER" -d "$DB_NAME" << 'EOF'
+-- Check migration status
+SELECT service, version, applied_at, description 
+FROM schema_migrations 
+WHERE service = 'requirement-processor' 
+ORDER BY applied_at;
+EOF
+
+log "✅ Migration fix completed!"
+log "You can now restart the full deployment:"
+log "docker compose up -d"

services/requirement-processor/migrations/001_business_context_tables.sql

@@ -7,7 +7,7 @@ CREATE TABLE IF NOT EXISTS business_context_responses (
     id UUID PRIMARY KEY DEFAULT uuid_generate_v4(),
     user_id UUID NOT NULL,
     template_id UUID,
-    project_id UUID REFERENCES projects(id) ON DELETE CASCADE,
+    project_id UUID,
     
     -- Simple JSONB structure with questions array
     questions JSONB NOT NULL DEFAULT '[]'::jsonb,

services/requirement-processor/migrations/migrate.py

@@ -16,8 +16,11 @@ DATABASE_URL = os.getenv('DATABASE_URL', 'postgresql://postgres:password@localho
 
 SCHEMA_MIGRATIONS_TABLE_SQL = """
 CREATE TABLE IF NOT EXISTS schema_migrations (
-  version TEXT PRIMARY KEY,
-  applied_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
+  id SERIAL PRIMARY KEY,
+  version VARCHAR(255) NOT NULL UNIQUE,
+  service VARCHAR(100) NOT NULL,
+  applied_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
+  description TEXT
 );
 """
 
@@ -25,20 +28,24 @@ async def ensure_migrations_table(pool) -> None:
     async with pool.acquire() as conn:
         await conn.execute(SCHEMA_MIGRATIONS_TABLE_SQL)
 
-async def is_applied(pool, version: str) -> bool:
+async def is_applied(pool, version: str, service: str = "requirement-processor") -> bool:
     async with pool.acquire() as conn:
-        row = await conn.fetchrow("SELECT 1 FROM schema_migrations WHERE version = $1", version)
+        row = await conn.fetchrow("SELECT 1 FROM schema_migrations WHERE version = $1 AND service = $2", version, service)
         return row is not None
 
-async def mark_applied(pool, version: str) -> None:
+async def mark_applied(pool, version: str, service: str = "requirement-processor", description: str = None) -> None:
     async with pool.acquire() as conn:
-        await conn.execute("INSERT INTO schema_migrations(version) VALUES($1) ON CONFLICT (version) DO NOTHING", version)
+        await conn.execute(
+            "INSERT INTO schema_migrations(version, service, description) VALUES($1, $2, $3) ON CONFLICT (version) DO NOTHING", 
+            version, service, description
+        )
 
 async def run_migration(pool, migration_file):
     """Run a single migration file if not applied"""
     version = migration_file.name
+    service = "requirement-processor"
     try:
-        if await is_applied(pool, version):
+        if await is_applied(pool, version, service):
             logger.info(f"⏭️  Skipping already applied migration: {version}")
             return True
 
@@ -48,7 +55,7 @@ async def run_migration(pool, migration_file):
         async with pool.acquire() as conn:
             await conn.execute(sql_content)
 
-        await mark_applied(pool, version)
+        await mark_applied(pool, version, service, f"Requirement processor migration: {version}")
         logger.info(f"✅ Migration completed: {version}")
         return True
     except Exception as e: