# Node.js Application Prompt: Dubai Land Department Analytics API

## Project Overview
Create a Node.js application that provides a REST API endpoint to process natural language queries about Dubai real estate data and return structured responses optimized for Chart.js visualization and frontend card displays.

## Technical Requirements

### Architecture
- **Clean Architecture**: Implement layered architecture with clear separation of concerns
- **Database**: MySQL with the provided Dubai Land Department schema
- **Response Format**: JSON responses compatible with Chart.js
- **Framework**: Express.js with TypeScript support
- **Database ORM**: Prisma or Sequelize for MySQL integration

### Core Functionality

#### 1. Natural Language Processing with Node.js NLP Libraries
- **Primary NLP Library**: Natural.js or Compromise.js for natural language understanding
- **Alternative Options**: 
  - **Natural.js**: Comprehensive NLP library with entity extraction, sentiment analysis, and classification
  - **Compromise.js**: Lightweight, fast NLP with excellent entity recognition
  - **Wink NLP**: Advanced NLP with custom entity recognition
  - **Node-nlp**: Microsoft's NLP library with intent recognition
- Accept user queries in natural language
- Parse queries using Node.js NLP libraries to identify:
  - Time periods (last 6 months, weekly, monthly)
  - Geographic areas (Business Bay, specific zones)
  - Property types (apartments, villas, commercial)
  - Aggregation types (trends, averages, summaries)
  - Transaction types (rental, sales, off-plan)
- Implement custom Named Entity Recognition (NER) for:
  - Dubai area names
  - Property type classification
  - Time period extraction
  - Metric type identification
- Use dependency parsing and tokenization for query structure analysis

#### 2. Query Processing Logic
The application should handle these specific query patterns:

**Single Value Queries** (Card Display):
- Calculate single metrics (averages, totals, counts)
- Return both descriptive text and executable SQL query
- Format results for frontend card components

**Multi-Query Scenarios** (Chart Visualization):
- Process complex queries requiring multiple data points
- Generate multiple SQL queries for comprehensive analysis
- Format results for Chart.js (line charts, bar charts, pie charts)

#### 3. Supported Query Types

**Rental Price Analysis:**
- "Give me the last 6 months rental price trend for Business Bay"
- "Summarise by week" (refinement of previous query)
- "Apartments only" (further refinement)

**Project Analysis:**
- "Brief about the Project" (transaction summary)
- "List of fast moving projects in last 6 months"
- "Which area is seeing uptick in off-plan projects in last 6 months"

**Area Performance:**
- "Which area is having more rental transactions?"
- "Top 5 areas for Commercial leasing and why?"
- "Top 5 areas for Residential leasing and why?"
- "Avg price of 3BHK apartment by area in last 6 months, group it by month. Show top 5 areas only."

## Database Schema Context

The application will work with the Dubai Land Department MySQL database containing:

### Core Tables:
- **transactions**: Real estate transaction records
- **rents**: Property rental contracts (Ejari system)
- **projects**: Development projects with developer relationships
- **developers**: Registered real estate developers
- **buildings**: Building registry information
- **lands**: Land registry information
- **valuations**: Official property valuations
- **brokers**: Registered real estate brokers

### Key Fields for Analysis:
- **Time Fields**: `instance_date`, `registration_date`, `start_date`, `end_date`
- **Location Fields**: `area_en`, `zone_en`, `nearest_metro_en`
- **Property Fields**: `prop_type_en`, `prop_sub_type_en`, `rooms_en`
- **Financial Fields**: `trans_value`, `contract_amount`, `annual_amount`
- **Project Fields**: `project_en`, `master_project_en`, `is_offplan_en`

## API Specification

### Endpoint Structure
```
POST /api/query
Content-Type: application/json

Request Body:
{
  "query": "Give me the last 6 months rental price trend for Business Bay"
}

Response Format:
{
  "success": true,
  "data": {
    "text": "Rental price trend for Business Bay over the last 6 months",
    "visualizations": [
      {
        "type": "line",
        "title": "Monthly Rental Price Trend",
        "data": {
          "labels": ["2024-01", "2024-02", "2024-03", "2024-04", "2024-05", "2024-06"],
          "datasets": [{
            "label": "Average Rental Price (AED)",
            "data": [85000, 87000, 89000, 92000, 95000, 98000],
            "borderColor": "rgb(75, 192, 192)",
            "backgroundColor": "rgba(75, 192, 192, 0.2)"
          }]
        }
      }
    ],
    "cards": [
      {
        "title": "Average Price",
        "value": "92,500 AED",
        "subtitle": "Last 6 months",
        "trend": "+15.3%"
      }
    ],
    "sql_queries": [
      "SELECT DATE_FORMAT(start_date, '%Y-%m') as month, AVG(annual_amount) as avg_price FROM rents WHERE area_en = 'Business Bay' AND start_date >= DATE_SUB(NOW(), INTERVAL 6 MONTH) GROUP BY DATE_FORMAT(start_date, '%Y-%m') ORDER BY month"
    ]
  }
}
```

## Implementation Requirements

### 1. Project Structure
```
src/
├── controllers/
│   └── queryController.js
├── services/
│   ├── nlpService.js          # spaCy integration and NLP processing
│   ├── queryParser.js         # Query parsing logic
│   ├── sqlGenerator.js        # SQL query generation
│   ├── chartFormatter.js      # Chart.js data formatting
│   └── contextManager.js      # Context tracking for follow-up queries
├── models/
│   └── database.js
├── middleware/
│   └── validation.js
├── utils/
│   ├── dateUtils.js
│   └── textProcessor.js
├── config/
│   ├── nlpConfig.js           # spaCy configuration and patterns
│   └── areaMapping.js         # Dubai area name mappings
└── routes/
    └── api.js
```

### 1.1. Required Dependencies

**Node.js Dependencies:**
```json
{
  "dependencies": {
    "express": "^4.18.2",
    "mysql2": "^3.6.0",
    "natural": "^6.5.0",
    "compromise": "^14.10.0",
    "wink-nlp": "^1.12.0",
    "node-nlp": "^4.27.0",
    "moment": "^2.29.4",
    "dotenv": "^16.3.1",
    "joi": "^17.11.0",
    "redis": "^4.6.0"
  },
  "devDependencies": {
    "@types/node": "^20.10.0",
    "typescript": "^5.3.3",
    "ts-node": "^10.9.1",
    "nodemon": "^3.0.2"
  }
}
```

**No Python Dependencies Required** - Pure Node.js implementation

### 2. Key Components

#### Query Parser Service (Node.js NLP-based)
- Load Natural.js or Compromise.js NLP library
- Extract time periods using custom regex patterns and date parsing
- Identify geographic areas using custom entity recognition
- Extract property types using keyword matching and classification
- Determine aggregation requirements using intent recognition
- Handle query refinements and follow-ups with context tracking
- Implement custom matcher patterns for Dubai-specific terminology
- Use tokenization and stemming for query normalization

#### SQL Generator Service
- Convert parsed queries to MySQL statements
- Handle date range calculations
- Implement proper JOIN operations for related tables
- Optimize queries for performance

#### Chart Formatter Service
- Convert SQL results to Chart.js compatible format
- Support multiple chart types (line, bar, pie, doughnut)
- Generate appropriate labels and datasets
- Handle data aggregation and grouping

### 3. Chart.js Compatibility

#### Line Charts (Trends)
```javascript
{
  type: 'line',
  data: {
    labels: ['Jan', 'Feb', 'Mar', 'Apr', 'May', 'Jun'],
    datasets: [{
      label: 'Rental Price Trend',
      data: [85000, 87000, 89000, 92000, 95000, 98000],
      borderColor: 'rgb(75, 192, 192)',
      backgroundColor: 'rgba(75, 192, 192, 0.2)'
    }]
  }
}
```

#### Bar Charts (Comparisons)
```javascript
{
  type: 'bar',
  data: {
    labels: ['Business Bay', 'Downtown', 'Marina', 'JBR', 'DIFC'],
    datasets: [{
      label: 'Average Rental Price',
      data: [92000, 85000, 78000, 95000, 110000],
      backgroundColor: 'rgba(54, 162, 235, 0.6)'
    }]
  }
}
```

#### Pie Charts (Distribution)
```javascript
{
  type: 'pie',
  data: {
    labels: ['Apartments', 'Villas', 'Commercial', 'Off-plan'],
    datasets: [{
      data: [45, 25, 20, 10],
      backgroundColor: ['#FF6384', '#36A2EB', '#FFCE56', '#4BC0C0']
    }]
  }
}
```

### 4. Node.js NLP Implementation Details

#### Option 1: Natural.js Implementation
```javascript
// Install Natural.js
// npm install natural
const natural = require('natural');
const moment = require('moment');

// Initialize NLP components
const tokenizer = new natural.WordTokenizer();
const stemmer = natural.PorterStemmer;

// Custom entity recognition for Dubai real estate
class DubaiRealEstateParser {
  constructor() {
    this.areaPatterns = [
      'Business Bay', 'Downtown', 'Marina', 'JBR', 'DIFC', 'JLT', 
      'Palm Jumeirah', 'Dubai Hills', 'Arabian Ranches', 'Jumeirah'
    ];
    
    this.propertyTypes = [
      'apartment', 'villa', 'commercial', 'office', 'retail', 'warehouse'
    ];
    
    this.roomTypes = ['studio', '1BHK', '2BHK', '3BHK', '4BHK', '5BHK'];
    
    this.intentKeywords = {
      trend: ['trend', 'over time', 'change', 'last', 'months', 'weekly', 'monthly'],
      compare: ['top', 'compare', 'versus', 'vs', 'better', 'best', 'highest', 'lowest'],
      average: ['average', 'avg', 'mean', 'typical', 'median'],
      summary: ['brief', 'summary', 'summarize', 'overview', 'overall'],
      count: ['how many', 'count', 'number of', 'total']
    };
  }

  async parseQuery(query) {
    const tokens = tokenizer.tokenize(query.toLowerCase());
    
    return {
      time_period: this.extractTimePeriod(query),
      areas: this.extractAreas(query),
      property_types: this.extractPropertyTypes(query),
      room_types: this.extractRoomTypes(query),
      intent: this.classifyIntent(query),
      tokens: tokens
    };
  }

  extractTimePeriod(query) {
    const timePatterns = [
      { pattern: /last (\d+) months?/i, value: 'months', count: 6 },
      { pattern: /last (\d+) weeks?/i, value: 'weeks', count: 4 },
      { pattern: /last (\d+) years?/i, value: 'years', count: 1 },
      { pattern: /(\d{4})/i, value: 'year', count: null }
    ];

    for (const timePattern of timePatterns) {
      const match = query.match(timePattern.pattern);
      if (match) {
        return {
          type: timePattern.value,
          count: match[1] ? parseInt(match[1]) : timePattern.count,
          startDate: this.calculateStartDate(timePattern.value, match[1] || timePattern.count)
        };
      }
    }
    return null;
  }

  extractAreas(query) {
    return this.areaPatterns.filter(area => 
      query.toLowerCase().includes(area.toLowerCase())
    );
  }

  extractPropertyTypes(query) {
    return this.propertyTypes.filter(type => 
      query.toLowerCase().includes(type.toLowerCase())
    );
  }

  extractRoomTypes(query) {
    return this.roomTypes.filter(room => 
      query.toLowerCase().includes(room.toLowerCase())
    );
  }

  classifyIntent(query) {
    const queryLower = query.toLowerCase();
    
    for (const [intent, keywords] of Object.entries(this.intentKeywords)) {
      if (keywords.some(keyword => queryLower.includes(keyword))) {
        return intent;
      }
    }
    return 'unknown';
  }

  calculateStartDate(period, count) {
    const now = moment();
    switch (period) {
      case 'months': return now.subtract(count, 'months').format('YYYY-MM-DD');
      case 'weeks': return now.subtract(count, 'weeks').format('YYYY-MM-DD');
      case 'years': return now.subtract(count, 'years').format('YYYY-MM-DD');
      default: return now.subtract(6, 'months').format('YYYY-MM-DD');
    }
  }
}
```

#### Option 2: Compromise.js Implementation
```javascript
// Install Compromise.js
// npm install compromise
const nlp = require('compromise');

class CompromiseRealEstateParser {
  constructor() {
    this.areaMapping = {
      'business bay': 'Business Bay',
      'downtown': 'Downtown Dubai',
      'marina': 'Dubai Marina',
      'jbr': 'JBR',
      'difc': 'DIFC'
    };
  }

  async parseQuery(query) {
    const doc = nlp(query);
    
    return {
      time_period: this.extractTimePeriod(doc),
      areas: this.extractAreas(doc),
      property_types: this.extractPropertyTypes(doc),
      intent: this.classifyIntent(doc),
      entities: doc.out('array')
    };
  }

  extractTimePeriod(doc) {
    const dates = doc.dates();
    const timeExpressions = doc.match('#Value+ (months?|weeks?|years?)');
    
    if (dates.length > 0) {
      return { type: 'specific_date', value: dates[0].text() };
    }
    
    if (timeExpressions.length > 0) {
      const match = timeExpressions.text().match(/(\d+)\s*(months?|weeks?|years?)/i);
      if (match) {
        return {
          type: match[2].replace('s', ''),
          count: parseInt(match[1])
        };
      }
    }
    
    return null;
  }

  extractAreas(doc) {
    const places = doc.places();
    const areas = [];
    
    places.forEach(place => {
      const placeText = place.text().toLowerCase();
      if (this.areaMapping[placeText]) {
        areas.push(this.areaMapping[placeText]);
      }
    });
    
    return areas;
  }

  extractPropertyTypes(doc) {
    const propertyKeywords = ['apartment', 'villa', 'commercial', 'office', 'retail'];
    return propertyKeywords.filter(keyword => 
      doc.has(keyword)
    );
  }

  classifyIntent(doc) {
    if (doc.has('trend') || doc.has('over time')) return 'trend';
    if (doc.has('top') || doc.has('best')) return 'compare';
    if (doc.has('average') || doc.has('avg')) return 'average';
    if (doc.has('summary') || doc.has('brief')) return 'summary';
    return 'unknown';
  }
}
```

#### Option 3: Wink NLP Implementation
```javascript
// Install Wink NLP
// npm install wink-nlp
const winkNLP = require('wink-nlp');
const model = require('wink-eng-lite-web-model');

const nlp = winkNLP(model);

class WinkRealEstateParser {
  constructor() {
    this.customEntities = {
      AREA: ['Business Bay', 'Downtown', 'Marina', 'JBR', 'DIFC'],
      PROPERTY_TYPE: ['apartment', 'villa', 'commercial', 'office'],
      ROOM_TYPE: ['studio', '1BHK', '2BHK', '3BHK', '4BHK']
    };
  }

  async parseQuery(query) {
    const doc = nlp.readDoc(query);
    
    return {
      time_period: this.extractTimePeriod(doc),
      areas: this.extractCustomEntities(doc, 'AREA'),
      property_types: this.extractCustomEntities(doc, 'PROPERTY_TYPE'),
      room_types: this.extractCustomEntities(doc, 'ROOM_TYPE'),
      intent: this.classifyIntent(doc),
      entities: doc.entities()
    };
  }

  extractTimePeriod(doc) {
    const dates = doc.entities().filter(e => e.type() === 'DATE');
    return dates.length > 0 ? dates[0].value() : null;
  }

  extractCustomEntities(doc, entityType) {
    const entities = [];
    this.customEntities[entityType].forEach(entity => {
      if (doc.has(entity)) {
        entities.push(entity);
      }
    });
    return entities;
  }

  classifyIntent(doc) {
    const intentPatterns = {
      trend: ['trend', 'over time', 'change'],
      compare: ['top', 'compare', 'best'],
      average: ['average', 'avg', 'mean'],
      summary: ['summary', 'brief', 'overview']
    };

    for (const [intent, keywords] of Object.entries(intentPatterns)) {
      if (keywords.some(keyword => doc.has(keyword))) {
        return intent;
      }
    }
    return 'unknown';
  }
}
```

#### Context Management for Follow-up Queries
```javascript
class QueryContextManager {
  constructor() {
    this.contexts = new Map(); // sessionId -> context
  }

  storeContext(sessionId, parsedQuery) {
    this.contexts.set(sessionId, {
      ...parsedQuery,
      timestamp: Date.now(),
      queryHistory: []
    });
  }

  getContext(sessionId) {
    return this.contexts.get(sessionId);
  }

  updateContext(sessionId, newQuery, isFollowUp = false) {
    const existingContext = this.getContext(sessionId);
    
    if (isFollowUp && existingContext) {
      // Merge with existing context
      const updatedContext = {
        ...existingContext,
        ...newQuery,
        refinements: this.extractRefinements(newQuery),
        timestamp: Date.now()
      };
      
      this.contexts.set(sessionId, updatedContext);
      return updatedContext;
    }
    
    // Store new context
    this.storeContext(sessionId, newQuery);
    return newQuery;
  }

  extractRefinements(query) {
    const refinements = [];
    
    if (query.includes('by week')) refinements.push('weekly_grouping');
    if (query.includes('by month')) refinements.push('monthly_grouping');
    if (query.includes('only')) refinements.push('filter_specific');
    if (query.includes('apartments only')) refinements.push('property_type_filter');
    
    return refinements;
  }

  isFollowUpQuery(query) {
    const followUpIndicators = [
      'by week', 'by month', 'only', 'filter', 'show', 
      'apartments only', 'villas only', 'commercial only'
    ];
    
    return followUpIndicators.some(indicator => 
      query.toLowerCase().includes(indicator)
    );
  }
}
```

### 5. Frontend Integration Support

#### Card Components
```javascript
{
  cards: [
    {
      title: "Average Rental Price",
      value: "92,500 AED",
      subtitle: "Business Bay - Last 6 months",
      trend: "+15.3%",
      icon: "trending-up"
    }
  ]
}
```

#### Visualization Decision Logic
- Single value results → Display as cards
- Time series data → Line charts
- Category comparisons → Bar charts
- Distribution data → Pie/doughnut charts
- Multiple metrics → Combination of cards and charts

## Development Guidelines

### 1. Error Handling
- Implement comprehensive error handling for SQL queries
- Provide meaningful error messages for invalid queries
- Handle database connection issues gracefully

### 2. Performance Optimization
- Use database indexes effectively
- Implement query result caching for common queries
- Optimize SQL queries for large datasets

### 3. Security
- Implement SQL injection prevention
- Validate and sanitize user inputs
- Use parameterized queries

### 4. Testing
- Unit tests for query parsing logic
- Integration tests for SQL generation
- API endpoint testing with sample queries

## Sample Implementation Flow

1. **Receive Query**: `"Give me the last 6 months rental price trend for Business Bay"`

2. **Process with Node.js NLP**:
   ```javascript
   const parser = new DubaiRealEstateParser();
   const parsed = await parser.parseQuery("Give me the last 6 months rental price trend for Business Bay");
   // Natural.js extracts:
   // - Time period: { type: 'months', count: 6, startDate: '2024-06-01' }
   // - Areas: ['Business Bay']
   // - Intent: 'trend'
   // - Property types: []
   ```

3. **Parse Query** (using Node.js NLP output):
   - Time period: Last 6 months (from time extraction)
   - Area: Business Bay (from area patterns)
   - Property type: All (not specified)
   - Analysis type: Trend (from intent classification)

4. **Generate SQL**:
   ```sql
   SELECT 
     DATE_FORMAT(start_date, '%Y-%m') as month,
     AVG(annual_amount) as avg_price,
     COUNT(*) as transaction_count
   FROM rents 
   WHERE area_en = 'Business Bay' 
     AND start_date >= DATE_SUB(NOW(), INTERVAL 6 MONTH)
   GROUP BY DATE_FORMAT(start_date, '%Y-%m')
   ORDER BY month
   ```

5. **Format Response**:
   - Create line chart data for trend visualization
   - Generate summary cards with key metrics
   - Provide descriptive text for the analysis

6. **Return JSON**: Structured response ready for Chart.js and frontend cards

## Environment Configuration

### Required Environment Variables
Create a `.env` file in the project root:
```env
# Server Configuration
PORT=3000
NODE_ENV=development

# Database Configuration
DB_HOST=localhost
DB_PORT=3306
DB_NAME=dubai_dld
DB_USER=root
DB_PASSWORD=your_password

# NLP Configuration
NLP_LIBRARY=natural
NLP_MODEL=en_core_web_sm

# Redis Configuration (optional - for context management)
REDIS_HOST=localhost
REDIS_PORT=6379
REDIS_PASSWORD=

# API Configuration
API_KEY=your_api_key_here
RATE_LIMIT=100
```

### Node.js NLP Service Configuration
```javascript
// config/nlpConfig.js
module.exports = {
  library: process.env.NLP_LIBRARY || 'natural',
  model: process.env.NLP_MODEL || 'en_core_web_sm',
  customPatterns: require('./customPatterns'),
  areaMapping: require('./areaMapping'),
  
  // Query classification keywords
  intentPatterns: {
    trend: ['trend', 'over time', 'change', 'last', 'months', 'weekly', 'monthly'],
    compare: ['top', 'compare', 'versus', 'vs', 'better', 'best', 'highest', 'lowest'],
    average: ['average', 'avg', 'mean', 'typical', 'median'],
    summary: ['brief', 'summary', 'summarize', 'overview', 'overall'],
    count: ['how many', 'count', 'number of', 'total'],
    filter: ['only', 'filter', 'show', 'exclude', 'include']
  }
};
```

## Additional Features

### Query Refinement Support with Node.js NLP Context Tracking
- Use Node.js NLP entity recognition to maintain context between queries
- Track extracted entities (areas, property types, time periods) across conversations
- Implement context-aware parsing for follow-up queries
- Store and retrieve previous query context from session management
- Support progressive data drilling with contextual refinements
- Use similarity matching for related area name variations

### Export Capabilities
- Provide raw data export options
- Support multiple output formats (CSV, JSON)
- Include metadata about queries and data sources

### Analytics Dashboard
- Track query patterns and popular analyses
- Monitor API usage and performance metrics
- Provide insights into data trends

This prompt provides a comprehensive foundation for building a sophisticated Node.js application that bridges natural language queries with structured data visualization, specifically tailored for Dubai real estate analytics.