517 lines
17 KiB
Markdown
517 lines
17 KiB
Markdown
# 🚀 Enhanced AI Tech Stack Selector v4.0
|
|
|
|
## 📋 Overview
|
|
|
|
The Enhanced AI Tech Stack Selector is an enterprise-grade, AI-powered system that intelligently analyzes business requirements and recommends optimal technology stacks. It's designed as part of a 4-service automated development pipeline that takes natural language requirements and outputs complete, production-ready applications.
|
|
|
|
## 🎯 System Purpose
|
|
|
|
**Input**: Processed requirements from the Requirement Processor (Port 8001)
|
|
**Output**: Intelligent technology stack recommendations with implementation roadmaps
|
|
**Integration**: Part of n8n workflow orchestration system
|
|
|
|
## 🏗️ Architecture Overview
|
|
|
|
```
|
|
┌─────────────────────────────────────────────────────────────────┐
|
|
│ Enhanced Tech Stack Selector │
|
|
├─────────────────────────────────────────────────────────────────┤
|
|
│ 🧠 Intelligence Layer │
|
|
│ ├── ContextOptimizationEngine (Token Management) │
|
|
│ ├── HallucinationPreventionEngine (Quality Assurance) │
|
|
│ ├── BusinessProblemAnalyzer (AI Business Understanding) │
|
|
│ ├── TechnologyIntelligenceEngine (Multi-AI Recommendations) │
|
|
│ └── EnhancedTechStackSelector (Main Orchestrator) │
|
|
├─────────────────────────────────────────────────────────────────┤
|
|
│ 💾 Storage Layer │
|
|
│ ├── Redis (Session Context - Fast Access) │
|
|
│ ├── PostgreSQL (Structured Decision History) │
|
|
│ ├── Neo4j (Technology Relationship Graphs) │
|
|
│ └── ChromaDB (Vector Similarity Search) │
|
|
├─────────────────────────────────────────────────────────────────┤
|
|
│ 🤖 AI Integration Layer │
|
|
│ ├── Claude 3.5 Sonnet (Primary Architecture Analysis) │
|
|
│ ├── GPT-4 Turbo (Secondary Validation) │
|
|
│ └── Rule-Based Engine (Baseline Validation) │
|
|
└─────────────────────────────────────────────────────────────────┘
|
|
```
|
|
|
|
## 🔧 Core Components
|
|
|
|
### 1. **ContextOptimizationEngine**
|
|
**Purpose**: Manages AI context within token limits while maximizing relevance
|
|
|
|
**Key Features**:
|
|
- **Token Budget Management**: Claude (180K), GPT-4 (100K), Local (8K)
|
|
- **Intelligent Context Selection**: Prioritizes most relevant information
|
|
- **Hierarchical Context Structure**: Level 1 (Critical) → Level 2 (Important) → Level 3 (Supporting)
|
|
- **Progressive Disclosure**: Adds context layers as needed
|
|
|
|
**How It Works**:
|
|
```python
|
|
# 1. Calculate available tokens
|
|
available_tokens = self.max_tokens['claude'] # 180,000
|
|
|
|
# 2. Gather all context components
|
|
context_components = {
|
|
'current_requirements': {'priority': 1.0, 'tokens': 5000},
|
|
'similar_decisions': {'priority': 0.9, 'tokens': 8000},
|
|
'technology_trends': {'priority': 0.7, 'tokens': 3000}
|
|
}
|
|
|
|
# 3. Select highest priority components that fit budget
|
|
selected_context = intelligent_selection(components, available_tokens)
|
|
|
|
# 4. Create hierarchical structure for progressive feeding
|
|
hierarchical_context = create_levels(selected_context)
|
|
```
|
|
|
|
### 2. **HallucinationPreventionEngine**
|
|
**Purpose**: Detects and prevents AI hallucinations in technology recommendations
|
|
|
|
**Validation Layers**:
|
|
- **Technology Existence**: Validates against known technology database
|
|
- **Scale Appropriateness**: Ensures tech choices match project scale
|
|
- **Domain Fit**: Validates domain-specific technology alignment
|
|
- **Internal Consistency**: Checks for contradictions in recommendations
|
|
- **Implementation Feasibility**: Validates team size vs complexity
|
|
|
|
**Technology Knowledge Base**:
|
|
```python
|
|
technology_knowledge_base = {
|
|
'frontend_frameworks': {
|
|
'react': {'maturity': 'high', 'ecosystem': 'excellent'},
|
|
'vue': {'maturity': 'high', 'ecosystem': 'good'},
|
|
'angular': {'maturity': 'high', 'ecosystem': 'excellent'}
|
|
},
|
|
'backend_technologies': {
|
|
'nodejs': {'performance': 'good', 'scalability': 'good'},
|
|
'python': {'performance': 'medium', 'scalability': 'good'},
|
|
'java': {'performance': 'excellent', 'scalability': 'excellent'}
|
|
}
|
|
}
|
|
```
|
|
|
|
### 3. **ContextPersistenceManager**
|
|
**Purpose**: Manages context storage across multiple database systems
|
|
|
|
**Storage Strategy**:
|
|
- **Redis**: Fast session context (1-hour TTL)
|
|
- **PostgreSQL**: Structured decision history with versioning
|
|
- **Neo4j**: Technology relationship graphs and patterns
|
|
- **ChromaDB**: Vector embeddings for semantic similarity search
|
|
|
|
**Data Flow**:
|
|
```
|
|
User Request → Context Retrieval → AI Analysis → Decision Storage
|
|
↓ ↓ ↓ ↓
|
|
Project ID → Redis Lookup → Multi-AI → All Databases
|
|
Session Cache Processing Updated
|
|
```
|
|
|
|
### 4. **BusinessProblemAnalyzer**
|
|
**Purpose**: Uses AI to understand core business problems dynamically
|
|
|
|
**Analysis Process**:
|
|
1. **Context Extraction**: Pulls domain, complexity, requirements from processor output
|
|
2. **AI Business Analysis**: Claude analyzes business model, value proposition, constraints
|
|
3. **Problem Characteristics**: Assesses complexity, scale, performance, team needs
|
|
4. **Fallback Logic**: Rule-based analysis when AI unavailable
|
|
|
|
**Business Model Detection**:
|
|
```python
|
|
# AI Prompt Example
|
|
"""
|
|
Analyze this business requirement:
|
|
Domain: ecommerce
|
|
Requirements: "Online marketplace for handmade crafts..."
|
|
|
|
Return JSON:
|
|
{
|
|
"core_business_problem": "Enable artisans to sell online",
|
|
"business_model": "marketplace",
|
|
"value_proposition": "Connect craft buyers with artisan sellers",
|
|
"success_criteria": ["vendor adoption", "transaction volume"]
|
|
}
|
|
"""
|
|
```
|
|
|
|
### 5. **TechnologyIntelligenceEngine**
|
|
**Purpose**: AI-driven technology recommendations with multi-model consensus
|
|
|
|
**Recommendation Process**:
|
|
1. **Context Optimization**: Prepare context for AI models
|
|
2. **Primary Analysis**: Claude generates comprehensive recommendations
|
|
3. **Secondary Validation**: GPT-4 validates and suggests improvements
|
|
4. **Multi-AI Consensus**: Synthesizes recommendations from multiple sources
|
|
5. **Final Assessment**: Risk analysis, implementation roadmap, success metrics
|
|
|
|
**AI Consensus Logic**:
|
|
```python
|
|
# Weight different AI models based on reliability
|
|
model_weights = {
|
|
'claude': 0.4, # Primary for architecture
|
|
'gpt4': 0.3, # Secondary for validation
|
|
'rule_based': 0.3 # Baseline validation
|
|
}
|
|
|
|
# Calculate consensus score
|
|
consensus_score = sum(model_confidence * weight for model, weight in weights)
|
|
```
|
|
|
|
### 6. **EnhancedTechStackSelector**
|
|
**Purpose**: Main orchestrator that coordinates all components
|
|
|
|
**Selection Process**:
|
|
1. **Context Retrieval**: Get conversation history for project continuity
|
|
2. **Business Analysis**: Understand the core business problem
|
|
3. **Historical Learning**: Find similar past decisions and success rates
|
|
4. **AI Recommendations**: Generate intelligent technology suggestions
|
|
5. **Validation & Enhancement**: Apply historical data and validation
|
|
6. **Response Generation**: Create comprehensive recommendation package
|
|
7. **Context Storage**: Store decision for future learning
|
|
|
|
## 🤖 AI Integration Details
|
|
|
|
### API Keys Configuration
|
|
```python
|
|
# Your actual API keys are configured in the code
|
|
CLAUDE_API_KEY = "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"
|
|
OPENAI_API_KEY = "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"
|
|
```
|
|
|
|
### Model Selection Strategy
|
|
- **Claude 3.5 Sonnet**: Primary model for architecture analysis (180K context)
|
|
- **GPT-4 Turbo**: Secondary validation and cross-checking (100K context)
|
|
- **Rule-Based**: Fallback when AI models unavailable
|
|
|
|
### Token Management
|
|
```python
|
|
# Progressive context feeding based on model capacity
|
|
if needs_more_context(response):
|
|
# Add Level 2 context
|
|
enhanced_prompt = base_prompt + level_2_context
|
|
response = call_ai_model(enhanced_prompt)
|
|
|
|
if still_needs_context(response):
|
|
# Add Level 3 context
|
|
final_prompt = enhanced_prompt + level_3_context
|
|
response = call_ai_model(final_prompt)
|
|
```
|
|
|
|
## 💾 Database Integration
|
|
|
|
### Redis (Session Context)
|
|
**Purpose**: Fast access to current conversation state
|
|
**TTL**: 1 hour
|
|
**Data Structure**:
|
|
```json
|
|
{
|
|
"last_analysis": {...},
|
|
"last_recommendations": {...},
|
|
"context_version": 2
|
|
}
|
|
```
|
|
|
|
### PostgreSQL (Structured Storage)
|
|
**Purpose**: Permanent storage of technology decisions
|
|
**Schema**:
|
|
```sql
|
|
CREATE TABLE tech_decisions (
|
|
id SERIAL PRIMARY KEY,
|
|
project_id VARCHAR(255) UNIQUE,
|
|
decision_data JSONB,
|
|
timestamp TIMESTAMP,
|
|
domain VARCHAR(100),
|
|
complexity VARCHAR(100),
|
|
version INTEGER DEFAULT 1
|
|
);
|
|
```
|
|
|
|
### Neo4j (Graph Relationships)
|
|
**Purpose**: Technology relationship patterns and domain connections
|
|
**Graph Structure**:
|
|
```
|
|
(Project)-[:HAS_DOMAIN]->(Domain)
|
|
(Project)-[:USES_FRONTEND]->(Frontend)
|
|
(Project)-[:USES_BACKEND]->(Backend)
|
|
(Frontend)-[:COMPATIBLE_WITH]->(Backend)
|
|
```
|
|
|
|
### ChromaDB (Vector Similarity)
|
|
**Purpose**: Semantic search for similar projects
|
|
**Process**:
|
|
1. Convert requirements to embeddings using SentenceTransformer
|
|
2. Store project embeddings with metadata
|
|
3. Query for similar projects using vector similarity
|
|
4. Return top 5 most similar past decisions
|
|
|
|
## 📡 API Endpoints
|
|
|
|
### Main Endpoint: `POST /api/v1/select`
|
|
**Purpose**: Primary technology stack selection endpoint
|
|
|
|
**Request Format**:
|
|
```json
|
|
{
|
|
"processed_requirements": {
|
|
"comprehensive_analysis": {...},
|
|
"original_requirements": "Build an e-commerce platform...",
|
|
"implementation_strategy": {...}
|
|
},
|
|
"project_name": "E-commerce Platform",
|
|
"project_id": "optional-for-context-continuity"
|
|
}
|
|
```
|
|
|
|
**Response Format**:
|
|
```json
|
|
{
|
|
"success": true,
|
|
"data": {
|
|
"project_id": "uuid-generated",
|
|
"analysis_metadata": {
|
|
"processing_method": "multi_model_consensus",
|
|
"confidence_score": 0.92,
|
|
"ai_models_used": ["claude", "openai", "rule_based"]
|
|
},
|
|
"business_problem_analysis": {...},
|
|
"technology_recommendations": {...},
|
|
"actionable_recommendations": {
|
|
"primary_stack": {...},
|
|
"implementation_priorities": [...],
|
|
"risk_mitigation_plan": {...}
|
|
}
|
|
}
|
|
}
|
|
```
|
|
|
|
### Health Check: `GET /health`
|
|
**Purpose**: System health monitoring
|
|
**Returns**: Component status, uptime, feature availability
|
|
|
|
### Debug Endpoints:
|
|
- `GET /api/v1/debug/ai-models` - Test AI model connectivity
|
|
- `GET /api/v1/context/{project_id}` - Retrieve project context
|
|
- `GET /api/v1/system-status` - Comprehensive system status
|
|
|
|
## 🔄 Processing Methods
|
|
|
|
The system supports multiple processing methods based on available resources:
|
|
|
|
### 1. **MULTI_MODEL_CONSENSUS** (Preferred)
|
|
- Uses Claude + GPT-4 + Rule-based analysis
|
|
- Highest confidence and accuracy
|
|
- Cross-validates recommendations
|
|
|
|
### 2. **CONTEXT_ENHANCED** (Single AI Model)
|
|
- Uses one AI model with enhanced context
|
|
- Good performance when only one model available
|
|
- Still includes validation layers
|
|
|
|
### 3. **RULE_BASED_ONLY** (Fallback)
|
|
- Pure rule-based analysis
|
|
- No AI models required
|
|
- Basic but functional recommendations
|
|
|
|
## 🛡️ Quality Assurance
|
|
|
|
### Hallucination Prevention
|
|
1. **Technology Validation**: Check against known technology database
|
|
2. **Consistency Checking**: Ensure internal logical consistency
|
|
3. **Scale Validation**: Match technology to project scale
|
|
4. **Domain Validation**: Ensure domain-appropriate choices
|
|
|
|
### Confidence Scoring
|
|
```python
|
|
# Multi-factor confidence calculation
|
|
base_confidence = ai_model_confidence # 0.9
|
|
validation_boost = validation_score # 0.85
|
|
historical_factor = success_rate # 0.8
|
|
|
|
final_confidence = (base_confidence * 0.5) +
|
|
(validation_boost * 0.3) +
|
|
(historical_factor * 0.2)
|
|
```
|
|
|
|
## 🚀 Development Setup
|
|
|
|
### Requirements
|
|
```bash
|
|
pip install fastapi uvicorn anthropic openai
|
|
pip install redis asyncpg neo4j chromadb
|
|
pip install sentence-transformers loguru
|
|
```
|
|
|
|
### Environment Variables
|
|
```bash
|
|
# API Keys (also hardcoded in main.py)
|
|
CLAUDE_API_KEY=your-claude-key
|
|
OPENAI_API_KEY=your-openai-key
|
|
|
|
# Database Connections
|
|
REDIS_HOST=redis
|
|
REDIS_PORT=6379
|
|
POSTGRES_URL=postgresql://user:pass@postgres:5432/db
|
|
NEO4J_URI=bolt://neo4j:7687
|
|
CHROMA_HOST=chromadb
|
|
CHROMA_PORT=8000
|
|
```
|
|
|
|
### Running the Service
|
|
```bash
|
|
# Development
|
|
python main.py
|
|
|
|
# Production
|
|
uvicorn main:app --host 0.0.0.0 --port 8002
|
|
|
|
# Docker
|
|
docker build -t tech-stack-selector .
|
|
docker run -p 8002:8002 tech-stack-selector
|
|
```
|
|
|
|
## 🔧 Integration with n8n Pipeline
|
|
|
|
### Pipeline Flow
|
|
```
|
|
User Input → Requirement Processor (8001) → Tech Stack Selector (8002) → Architecture Designer (8003) → Code Generator (8004)
|
|
```
|
|
|
|
### n8n Configuration
|
|
```json
|
|
{
|
|
"name": "Tech Stack Selection",
|
|
"type": "HTTP Request",
|
|
"url": "http://tech-stack-selector:8002/api/v1/select",
|
|
"method": "POST",
|
|
"body": "{{ $json.data }}"
|
|
}
|
|
```
|
|
|
|
## 📊 Monitoring & Debugging
|
|
|
|
### Health Monitoring
|
|
- Component health checks for all databases
|
|
- AI model connectivity testing
|
|
- Feature availability status
|
|
|
|
### Logging
|
|
- Structured logging with loguru
|
|
- Request/response logging
|
|
- Error tracking and debugging
|
|
- Performance metrics
|
|
|
|
### Debug Tools
|
|
- AI model connectivity testing
|
|
- Context retrieval and inspection
|
|
- System status comprehensive view
|
|
- Storage system health checks
|
|
|
|
## 🎯 Future Enhancement Opportunities
|
|
|
|
### For Junior Developers
|
|
|
|
1. **Enhanced Business Logic**
|
|
- Add more domain-specific patterns
|
|
- Improve complexity scoring algorithms
|
|
- Add industry-specific recommendations
|
|
|
|
2. **AI Model Improvements**
|
|
- Add more AI models (Gemini, etc.)
|
|
- Implement custom fine-tuned models
|
|
- Add specialized domain models
|
|
|
|
3. **Context Optimization**
|
|
- Implement more sophisticated embedding models
|
|
- Add semantic chunking algorithms
|
|
- Improve relevance scoring
|
|
|
|
4. **Storage Enhancements**
|
|
- Add time-series analysis
|
|
- Implement better caching strategies
|
|
- Add backup and recovery systems
|
|
|
|
5. **API Improvements**
|
|
- Add streaming responses
|
|
- Implement webhooks for updates
|
|
- Add batch processing capabilities
|
|
|
|
## 🐛 Common Issues & Solutions
|
|
|
|
### Issue: AI Model Not Responding
|
|
**Symptoms**: 500 errors, timeout responses
|
|
**Solution**: Check API keys, test connectivity via debug endpoint
|
|
|
|
### Issue: Context Not Persisting
|
|
**Symptoms**: No conversation history, recommendations not improving
|
|
**Solution**: Verify database connections, check Redis TTL settings
|
|
|
|
### Issue: Low Confidence Scores
|
|
**Symptoms**: Confidence < 0.7, validation warnings
|
|
**Solution**: Check input quality, verify AI model responses, review validation rules
|
|
|
|
### Issue: Poor Recommendations
|
|
**Symptoms**: Inappropriate technology choices, mismatched scale
|
|
**Solution**: Review business problem analysis, check domain classification, verify complexity scoring
|
|
|
|
## 📝 Code Examples
|
|
|
|
### Adding a New Domain
|
|
```python
|
|
# In BusinessProblemAnalyzer._fallback_business_analysis()
|
|
elif 'gaming' in domain or 'game' in requirements:
|
|
business_model = "gaming"
|
|
core_problem = "Create engaging gaming experience"
|
|
|
|
# In ContextOptimizationEngine._get_business_indicators()
|
|
'gaming': ['real-time', 'multiplayer', 'graphics', 'performance']
|
|
```
|
|
|
|
### Adding Custom Validation Rules
|
|
```python
|
|
# In HallucinationPreventionEngine._build_validation_rules()
|
|
'gaming_validation': {
|
|
'required_features': ['real_time', 'graphics', 'performance'],
|
|
'recommended_tech': ['unity', 'unreal', 'webgl']
|
|
}
|
|
```
|
|
|
|
### Extending AI Prompts
|
|
```python
|
|
# In TechnologyIntelligenceEngine._build_context_optimized_prompt()
|
|
if domain == 'gaming':
|
|
base_prompt += """
|
|
## Gaming-Specific Considerations:
|
|
- Real-time performance requirements
|
|
- Graphics and rendering needs
|
|
- Multiplayer architecture considerations
|
|
"""
|
|
```
|
|
|
|
## 📚 Additional Resources
|
|
|
|
- **FastAPI Documentation**: https://fastapi.tiangolo.com/
|
|
- **Claude API**: https://docs.anthropic.com/
|
|
- **OpenAI API**: https://platform.openai.com/docs
|
|
- **Neo4j Documentation**: https://neo4j.com/docs/
|
|
- **ChromaDB Guide**: https://docs.trychroma.com/
|
|
|
|
---
|
|
|
|
**Last Updated**: July 3, 2025
|
|
**Version**: 4.0.0
|
|
**Maintainer**: AI Development Pipeline Team
|
|
**Status**: Production Ready ✅
|
|
|
|
# Normal startup (auto-detects if migration needed)
|
|
./start_migrated.sh
|
|
|
|
# Force re-migration (useful when you add new data)
|
|
./start_migrated.sh --force-migration
|
|
|
|
# Show help
|
|
./start_migrated.sh --help
|
|
|
|
|
|
healthcare, finance, gaming, education, media, iot, social, elearning, realestate, travel, manufacturing, ecommerce, saas |