codenuk_backend_mine/DATABASE_MIGRATION_FIX.md

Database Migration Issues - SOLVED

Problem Summary

You were experiencing unwanted tables being created and duplicates when starting the server. This was caused by multiple migration sources creating the same tables and conflicting migration execution.

Root Causes Identified

1. Multiple Migration Sources

• PostgreSQL init script (databases/scripts/init.sql) creates the dev_pipeline database
• Shared schemas (databases/scripts/schemas.sql) creates core tables
• Individual service migrations create their own tables
• Template-manager was also applying shared schemas, causing duplicates

2. Migration Execution Order Issues

• Services were running migrations in parallel
• No proper dependency management between shared schemas and service-specific tables
• DROP TABLE statements in development mode causing data loss

3. Table Conflicts

• users table created by both schemas.sql and user-auth migration
• user_projects table created by both sources
• Function conflicts (update_updated_at_column() created multiple times)
• Extension conflicts (uuid-ossp created multiple times)

Solutions Implemented

1. Fixed Migration Order

• Created separate shared-schemas service for core database tables
• Updated migration script to run in correct order:
  1. shared-schemas (core tables first)
  2. user-auth (user-specific tables)
  3. template-manager (template-specific tables)

2. Made Migrations Production-Safe

• Replaced DROP TABLE IF EXISTS with CREATE TABLE IF NOT EXISTS
• Prevents data loss on server restarts
• Safe for production environments

3. Eliminated Duplicate Table Creation

• Removed shared schema application from template-manager
• Each service now only creates its own tables
• Proper dependency management

4. Created Database Cleanup Script

• scripts/cleanup-database.sh removes unwanted/duplicate tables
• Can be run to clean up existing database issues

How to Use

Clean Up Existing Database

cd /home/tech4biz/Desktop/Projectsnew/CODENUK1/codenuk-backend-live
./scripts/cleanup-database.sh

Start Server with Fixed Migrations

docker-compose up --build

The migrations will now run in the correct order:

1. Shared schemas (projects, tech_stack_decisions, etc.)
2. User authentication tables
3. Template management tables

Files Modified

1. services/template-manager/src/migrations/migrate.js

   • Removed shared schema application
   • Now only handles template-specific tables

2. services/user-auth/src/migrations/001_user_auth_schema.sql

   • Replaced DROP TABLE with CREATE TABLE IF NOT EXISTS
   • Made migration production-safe

3. services/template-manager/src/migrations/001_initial_schema.sql

   • Replaced DROP TABLE with CREATE TABLE IF NOT EXISTS
   • Made migration production-safe

4. scripts/migrate-all.sh

   • Added shared-schemas service
   • Proper migration order

5. docker-compose.yml

   • Removed APPLY_SCHEMAS_SQL environment variable

6. Created new files:

   • services/shared-schemas/ - Dedicated service for shared schemas
   • scripts/cleanup-database.sh - Database cleanup script

Expected Results

After these changes:

• ✅ No duplicate tables will be created
• ✅ No unwanted tables from pgAdmin
• ✅ Proper migration order
• ✅ Production-safe migrations
• ✅ Clean database schema

Verification

To verify the fix worked:

1. Run the cleanup script
2. Start the server
3. Check pgAdmin - you should only see the intended tables
4. No duplicate or unwanted tables should appear