codenuk_backend_mine/services/template-manager/TKG_MIGRATION_README.md

Template Knowledge Graph (TKG) Migration System

Overview

The Template Knowledge Graph (TKG) migration system migrates data from PostgreSQL to Neo4j to create a comprehensive knowledge graph that maps:

• Templates → Features → Technologies
• Tech Stack Recommendations → Technologies by Category
• Feature Dependencies and Technology Synergies

Architecture

1. Neo4j Graph Structure

Template → HAS_FEATURE → Feature → REQUIRES_TECHNOLOGY → Technology
    ↓
HAS_TECH_STACK → TechStack → RECOMMENDS_TECHNOLOGY → Technology

2. Node Types

• Template: Application templates (e-commerce, SaaS, etc.)
• Feature: Individual features (authentication, payment, etc.)
• Technology: Tech stack components (React, Node.js, etc.)
• TechStack: AI-generated tech stack recommendations

3. Relationship Types

• HAS_FEATURE: Template contains feature
• REQUIRES_TECHNOLOGY: Feature needs technology
• RECOMMENDS_TECHNOLOGY: Tech stack recommends technology
• HAS_TECH_STACK: Template has tech stack

API Endpoints

Migration Endpoints

• POST /api/tkg-migration/migrate - Migrate all data to TKG
• GET /api/tkg-migration/stats - Get migration statistics
• POST /api/tkg-migration/clear - Clear TKG data
• GET /api/tkg-migration/health - Health check

Template Endpoints

• POST /api/tkg-migration/template/:id - Migrate single template
• GET /api/tkg-migration/template/:id/tech-stack - Get template tech stack
• GET /api/tkg-migration/template/:id/features - Get template features

Usage

1. Start the Service

cd services/template-manager
npm start

2. Run Migration

# Full migration
curl -X POST http://localhost:8009/api/tkg-migration/migrate

# Get stats
curl http://localhost:8009/api/tkg-migration/stats

# Health check
curl http://localhost:8009/api/tkg-migration/health

3. Test Migration

node test/test-tkg-migration.js

Configuration

Environment Variables

# Neo4j Configuration
NEO4J_URI=bolt://localhost:7687
NEO4J_USERNAME=neo4j
NEO4J_PASSWORD=password

# Database Configuration
DB_HOST=localhost
DB_PORT=5432
DB_NAME=template_manager
DB_USER=postgres
DB_PASSWORD=password

Migration Process

1. Data Sources

• Templates: From templates and custom_templates tables
• Features: From features and custom_features tables
• Tech Stack: From tech_stack_recommendations table

2. Migration Steps

1. Clear existing Neo4j data
2. Migrate default templates with features
3. Migrate custom templates with features
4. Migrate tech stack recommendations
5. Create technology relationships
6. Generate migration statistics

3. AI-Powered Analysis

The system uses Claude AI to:

• Extract technologies from feature descriptions
• Analyze business rules for tech requirements
• Generate technology confidence scores
• Identify feature dependencies

Neo4j Queries

Get Template Tech Stack

MATCH (t:Template {id: $templateId})
MATCH (t)-[:HAS_TECH_STACK]->(ts)
MATCH (ts)-[r:RECOMMENDS_TECHNOLOGY]->(tech)
RETURN ts, tech, r.category, r.confidence
ORDER BY r.category, r.confidence DESC

Get Template Features

MATCH (t:Template {id: $templateId})
MATCH (t)-[:HAS_FEATURE]->(f)
MATCH (f)-[:REQUIRES_TECHNOLOGY]->(tech)
RETURN f, tech
ORDER BY f.display_order, f.name

Get Technology Synergies

MATCH (tech1:Technology)-[:SYNERGY]->(tech2:Technology)
RETURN tech1.name, tech2.name, synergy_score
ORDER BY synergy_score DESC

Error Handling

The migration system includes comprehensive error handling:

• Connection failures: Graceful fallback to PostgreSQL
• Data validation: Skip invalid records with logging
• Partial failures: Continue migration with error reporting
• Rollback support: Clear and retry functionality

Performance Considerations

• Batch processing: Migrate templates in batches
• Connection pooling: Reuse Neo4j connections
• Indexing: Create indexes on frequently queried properties
• Memory management: Close connections properly

Monitoring

Migration Statistics

• Templates migrated
• Features migrated
• Technologies created
• Tech stacks migrated
• Relationships created

Health Monitoring

• Neo4j connection status
• Migration progress
• Error rates
• Performance metrics

Troubleshooting

Common Issues

1. Neo4j connection failed

   • Check Neo4j service status
   • Verify connection credentials
   • Ensure Neo4j is running on correct port

2. Migration timeout

   • Increase timeout settings
   • Check Neo4j memory settings
   • Monitor system resources

3. Data validation errors

   • Check PostgreSQL data integrity
   • Verify required fields are present
   • Review migration logs

Debug Commands

# Check Neo4j status
docker ps | grep neo4j

# View Neo4j logs
docker logs neo4j-container

# Test Neo4j connection
cypher-shell -u neo4j -p password "RETURN 1"

Future Enhancements

1. Incremental Migration: Only migrate changed data
2. Real-time Sync: Keep Neo4j in sync with PostgreSQL
3. Advanced Analytics: Technology trend analysis
4. Recommendation Engine: AI-powered tech stack suggestions
5. Visualization: Graph visualization tools

Support

For issues or questions:

1. Check the logs for error messages
2. Verify Neo4j and PostgreSQL connections
3. Review migration statistics
4. Test with single template migration first