codenuk_backend_mine/DATABASE_MIGRATION_CLEAN.md

Database Migration System - Clean & Organized

Overview

This document explains the new clean database migration system that resolves the issues with unwanted tables and duplicate table creation.

Problems Solved

❌ Previous Issues

• Duplicate tables: Multiple services creating the same tables (users, user_projects, etc.)
• Unwanted tables: Tech-stack-selector creating massive schema with 100+ tables
• Inconsistent migrations: Some services using DROP TABLE, others using CREATE TABLE IF NOT EXISTS
• Missing shared-schemas: Migration script referenced non-existent service
• AI-mockup-service duplication: Creating same tables as user-auth service

✅ Solutions Implemented

1. Clean Database Reset: Complete schema reset before applying migrations
2. Proper Migration Order: Core schema first, then service-specific tables
3. Minimal Service Schemas: Each service only creates tables it actually needs
4. Consistent Approach: All services use CREATE TABLE IF NOT EXISTS
5. Migration Tracking: Proper tracking of applied migrations

Migration System Architecture

1. Core Schema (databases/scripts/schemas.sql)

Tables Created:

• projects - Main project tracking
• tech_stack_decisions - Technology choices per project
• system_architectures - Architecture designs
• code_generations - Generated code tracking
• test_results - Test execution results
• deployment_logs - Deployment tracking
• service_health - Service monitoring
• project_state_transitions - Audit trail

2. Service-Specific Tables

User Authentication Service (user-auth)

Tables Created:

• users - User accounts
• refresh_tokens - JWT refresh tokens
• user_sessions - User session tracking
• user_feature_preferences - Feature customization
• user_projects - User project tracking

Template Manager Service (template-manager)

Tables Created:

• templates - Template definitions
• template_features - Feature definitions
• feature_usage - Usage tracking
• custom_features - User-created features

Requirement Processor Service (requirement-processor)

Tables Created:

• business_context_responses - Business context data
• question_templates - Reusable question sets

Git Integration Service (git-integration)

Tables Created:

• github_repositories - Repository tracking
• github_user_tokens - OAuth tokens
• repository_storage - Local storage tracking
• repository_directories - Directory structure
• repository_files - File tracking

AI Mockup Service (ai-mockup-service)

Tables Created:

• wireframes - Wireframe data
• wireframe_versions - Version tracking
• wireframe_elements - Element analysis

Tech Stack Selector Service (tech-stack-selector)

Tables Created:

• tech_stack_recommendations - AI recommendations
• stack_analysis_cache - Analysis caching

How to Use

Clean Database Migration

cd /home/tech4biz/Desktop/Projectsnew/CODENUK1/codenuk-backend-live

# Run the clean migration script
./scripts/migrate-clean.sh

Start Services with Clean Database

# Start all services with clean migrations
docker-compose up --build

# Or start specific services
docker-compose up postgres redis migrations

Manual Database Cleanup (if needed)

# Run the cleanup script to remove unwanted tables
./scripts/cleanup-database.sh

Migration Process

Step 1: Database Cleanup

• Drops all existing tables
• Recreates public schema
• Re-enables required extensions
• Creates migration tracking table

Step 2: Core Schema Application

• Applies databases/scripts/schemas.sql
• Creates core pipeline tables
• Marks as applied in migration tracking

Step 3: Service Migrations

• Runs migrations in dependency order:
  1. user-auth (user tables first)
  2. template-manager (template tables)
  3. requirement-processor (business context)
  4. git-integration (repository tracking)
  5. ai-mockup-service (wireframe tables)
  6. tech-stack-selector (recommendation tables)

Step 4: Verification

• Lists all created tables
• Shows applied migrations
• Confirms successful completion

Service Migration Scripts

Node.js Services

• user-auth: npm run migrate
• template-manager: npm run migrate
• git-integration: npm run migrate

Python Services

• ai-mockup-service: python3 src/migrations/migrate.py
• tech-stack-selector: python3 migrate.py
• requirement-processor: python3 migrations/migrate.py

Expected Final Tables

After running the clean migration, you should see these tables:

Core Tables (8)

• projects
• tech_stack_decisions
• system_architectures
• code_generations
• test_results
• deployment_logs
• service_health
• project_state_transitions

User Auth Tables (5)

• users
• refresh_tokens
• user_sessions
• user_feature_preferences
• user_projects

Template Manager Tables (4)

• templates
• template_features
• feature_usage
• custom_features

Requirement Processor Tables (2)

• business_context_responses
• question_templates

Git Integration Tables (5)

• github_repositories
• github_user_tokens
• repository_storage
• repository_directories
• repository_files

AI Mockup Tables (3)

• wireframes
• wireframe_versions
• wireframe_elements

Tech Stack Selector Tables (2)

• tech_stack_recommendations
• stack_analysis_cache

System Tables (1)

• schema_migrations

Total: 29 tables (vs 100+ previously)

Troubleshooting

If Migration Fails

1. Check database connection parameters
2. Ensure all required extensions are available
3. Verify service directories exist
4. Check migration script permissions

If Unwanted Tables Appear

1. Run ./scripts/cleanup-database.sh
2. Restart with docker-compose up --build
3. Check service migration scripts for DROP statements

If Services Don't Start

1. Check migration dependencies in docker-compose.yml
2. Verify migration script completed successfully
3. Check service logs for database connection issues

Benefits

✅ Clean Database: Only necessary tables created ✅ No Duplicates: Each table created by one service only ✅ Proper Dependencies: Tables created in correct order ✅ Production Safe: Uses CREATE TABLE IF NOT EXISTS ✅ Trackable: All migrations tracked and logged ✅ Maintainable: Clear separation of concerns ✅ Scalable: Easy to add new services

Next Steps

1. Test the migration: Run ./scripts/migrate-clean.sh
2. Start services: Run docker-compose up --build
3. Verify tables: Check pgAdmin for clean table list
4. Monitor logs: Ensure all services start successfully

The database is now clean, organized, and ready for production use!