codenuk_backend_mine/services/template-manager/TKG_MIGRATION_README.md

# Template Knowledge Graph (TKG) Migration System

## Overview

The Template Knowledge Graph (TKG) migration system migrates data from PostgreSQL to Neo4j to create a comprehensive knowledge graph that maps:

- **Templates** → **Features** → **Technologies**
- **Tech Stack Recommendations** → **Technologies by Category**
- **Feature Dependencies** and **Technology Synergies**

## Architecture

### 1. Neo4j Graph Structure

```
Template → HAS_FEATURE → Feature → REQUIRES_TECHNOLOGY → Technology
    ↓
HAS_TECH_STACK → TechStack → RECOMMENDS_TECHNOLOGY → Technology
```

### 2. Node Types

- **Template**: Application templates (e-commerce, SaaS, etc.)
- **Feature**: Individual features (authentication, payment, etc.)
- **Technology**: Tech stack components (React, Node.js, etc.)
- **TechStack**: AI-generated tech stack recommendations

### 3. Relationship Types

- **HAS_FEATURE**: Template contains feature
- **REQUIRES_TECHNOLOGY**: Feature needs technology
- **RECOMMENDS_TECHNOLOGY**: Tech stack recommends technology
- **HAS_TECH_STACK**: Template has tech stack

## API Endpoints

### Migration Endpoints

- `POST /api/tkg-migration/migrate` - Migrate all data to TKG
- `GET /api/tkg-migration/stats` - Get migration statistics
- `POST /api/tkg-migration/clear` - Clear TKG data
- `GET /api/tkg-migration/health` - Health check

### Template Endpoints

- `POST /api/tkg-migration/template/:id` - Migrate single template
- `GET /api/tkg-migration/template/:id/tech-stack` - Get template tech stack
- `GET /api/tkg-migration/template/:id/features` - Get template features

## Usage

### 1. Start the Service

```bash
cd services/template-manager
npm start
```

### 2. Run Migration

```bash
# Full migration
curl -X POST http://localhost:8009/api/tkg-migration/migrate

# Get stats
curl http://localhost:8009/api/tkg-migration/stats

# Health check
curl http://localhost:8009/api/tkg-migration/health
```

### 3. Test Migration

```bash
node test/test-tkg-migration.js
```

## Configuration

### Environment Variables

```bash
# Neo4j Configuration
NEO4J_URI=bolt://localhost:7687
NEO4J_USERNAME=neo4j
NEO4J_PASSWORD=password

# Database Configuration
DB_HOST=localhost
DB_PORT=5432
DB_NAME=template_manager
DB_USER=postgres
DB_PASSWORD=password
```

## Migration Process

### 1. Data Sources

- **Templates**: From `templates` and `custom_templates` tables
- **Features**: From `features` and `custom_features` tables
- **Tech Stack**: From `tech_stack_recommendations` table

### 2. Migration Steps

1. **Clear existing Neo4j data**
2. **Migrate default templates** with features
3. **Migrate custom templates** with features
4. **Migrate tech stack recommendations**
5. **Create technology relationships**
6. **Generate migration statistics**

### 3. AI-Powered Analysis

The system uses Claude AI to:
- Extract technologies from feature descriptions
- Analyze business rules for tech requirements
- Generate technology confidence scores
- Identify feature dependencies

## Neo4j Queries

### Get Template Tech Stack

```cypher
MATCH (t:Template {id: $templateId})
MATCH (t)-[:HAS_TECH_STACK]->(ts)
MATCH (ts)-[r:RECOMMENDS_TECHNOLOGY]->(tech)
RETURN ts, tech, r.category, r.confidence
ORDER BY r.category, r.confidence DESC
```

### Get Template Features

```cypher
MATCH (t:Template {id: $templateId})
MATCH (t)-[:HAS_FEATURE]->(f)
MATCH (f)-[:REQUIRES_TECHNOLOGY]->(tech)
RETURN f, tech
ORDER BY f.display_order, f.name
```

### Get Technology Synergies

```cypher
MATCH (tech1:Technology)-[:SYNERGY]->(tech2:Technology)
RETURN tech1.name, tech2.name, synergy_score
ORDER BY synergy_score DESC
```

## Error Handling

The migration system includes comprehensive error handling:

- **Connection failures**: Graceful fallback to PostgreSQL
- **Data validation**: Skip invalid records with logging
- **Partial failures**: Continue migration with error reporting
- **Rollback support**: Clear and retry functionality

## Performance Considerations

- **Batch processing**: Migrate templates in batches
- **Connection pooling**: Reuse Neo4j connections
- **Indexing**: Create indexes on frequently queried properties
- **Memory management**: Close connections properly

## Monitoring

### Migration Statistics

- Templates migrated
- Features migrated
- Technologies created
- Tech stacks migrated
- Relationships created

### Health Monitoring

- Neo4j connection status
- Migration progress
- Error rates
- Performance metrics

## Troubleshooting

### Common Issues

1. **Neo4j connection failed**
   - Check Neo4j service status
   - Verify connection credentials
   - Ensure Neo4j is running on correct port

2. **Migration timeout**
   - Increase timeout settings
   - Check Neo4j memory settings
   - Monitor system resources

3. **Data validation errors**
   - Check PostgreSQL data integrity
   - Verify required fields are present
   - Review migration logs

### Debug Commands

```bash
# Check Neo4j status
docker ps | grep neo4j

# View Neo4j logs
docker logs neo4j-container

# Test Neo4j connection
cypher-shell -u neo4j -p password "RETURN 1"
```

## Future Enhancements

1. **Incremental Migration**: Only migrate changed data
2. **Real-time Sync**: Keep Neo4j in sync with PostgreSQL
3. **Advanced Analytics**: Technology trend analysis
4. **Recommendation Engine**: AI-powered tech stack suggestions
5. **Visualization**: Graph visualization tools

## Support

For issues or questions:
1. Check the logs for error messages
2. Verify Neo4j and PostgreSQL connections
3. Review migration statistics
4. Test with single template migration first