Re_Backend/docs/TAT_NOTIFICATION_SYSTEM.md

TAT (Turnaround Time) Notification System

Overview

The TAT Notification System automatically tracks and notifies approvers about their approval deadlines at key milestones (50%, 75%, and 100% of allotted time). It uses a queue-based architecture with BullMQ and Redis to ensure reliable, scheduled notifications.

Architecture

┌─────────────────┐
│  Workflow       │
│  Submission     │
└────────┬────────┘
         │
         ├──> Schedule TAT Jobs (50%, 75%, 100%)
         │
┌────────▼────────┐     ┌──────────────┐     ┌─────────────┐
│   TAT Queue     │────>│  TAT Worker  │────>│ Processor   │
│   (BullMQ)      │     │  (Background)│     │  Handler    │
└─────────────────┘     └──────────────┘     └──────┬──────┘
                                                     │
                                                     ├──> Send Notification
                                                     ├──> Update Database
                                                     └──> Log Activity

Components

1. TAT Time Utilities (tatTimeUtils.ts)

Handles working hours calculations (Monday-Friday, 9 AM - 6 PM):

// Calculate TAT milestones considering working hours
const { halfTime, seventyFive, full } = calculateTatMilestones(startDate, tatHours);

Key Functions:

• addWorkingHours(): Adds working hours to a start date, skipping weekends
• calculateTatMilestones(): Calculates 50%, 75%, and 100% time points
• calculateDelay(): Computes delay in milliseconds from now to target

2. TAT Queue (tatQueue.ts)

BullMQ queue configuration with Redis:

export const tatQueue = new Queue('tatQueue', {
  connection: IORedis,
  defaultJobOptions: {
    removeOnComplete: true,
    removeOnFail: false,
    attempts: 3,
    backoff: { type: 'exponential', delay: 2000 }
  }
});

3. TAT Processor (tatProcessor.ts)

Handles job execution when TAT milestones are reached:

export async function handleTatJob(job: Job<TatJobData>) {
  // Process tat50, tat75, or tatBreach
  // - Send notification to approver
  // - Update database flags
  // - Log activity
}

Job Types:

• tat50: ⏳ 50% of TAT elapsed (gentle reminder)
• tat75: ⚠️ 75% of TAT elapsed (escalation warning)
• tatBreach: ⏰ 100% of TAT elapsed (breach notification)

4. TAT Worker (tatWorker.ts)

Background worker that processes jobs from the queue:

export const tatWorker = new Worker('tatQueue', handleTatJob, {
  connection,
  concurrency: 5,
  limiter: { max: 10, duration: 1000 }
});

Features:

• Concurrent job processing (up to 5 jobs)
• Rate limiting (10 jobs/second)
• Automatic retry on failure
• Graceful shutdown on SIGTERM/SIGINT

5. TAT Scheduler Service (tatScheduler.service.ts)

Service for scheduling and managing TAT jobs:

// Schedule TAT jobs for an approval level
await tatSchedulerService.scheduleTatJobs(
  requestId,
  levelId,
  approverId,
  tatHours,
  startTime
);

// Cancel TAT jobs when level is completed
await tatSchedulerService.cancelTatJobs(requestId, levelId);

Database Schema

New Fields in approval_levels Table

ALTER TABLE approval_levels ADD COLUMN tat50_alert_sent BOOLEAN NOT NULL DEFAULT false;
ALTER TABLE approval_levels ADD COLUMN tat75_alert_sent BOOLEAN NOT NULL DEFAULT false;
ALTER TABLE approval_levels ADD COLUMN tat_breached BOOLEAN NOT NULL DEFAULT false;
ALTER TABLE approval_levels ADD COLUMN tat_start_time TIMESTAMP WITH TIME ZONE;

Field Descriptions:

• tat50_alert_sent: Tracks if 50% notification was sent
• tat75_alert_sent: Tracks if 75% notification was sent
• tat_breached: Tracks if TAT deadline was breached
• tat_start_time: Timestamp when TAT monitoring started

Integration Points

1. Workflow Submission

When a workflow is submitted, TAT monitoring starts for the first approval level:

// workflow.service.ts - submitWorkflow()
await current.update({
  levelStartTime: now,
  tatStartTime: now,
  status: ApprovalStatus.IN_PROGRESS
});

await tatSchedulerService.scheduleTatJobs(
  requestId,
  levelId,
  approverId,
  tatHours,
  now
);

2. Approval Flow

When a level is approved, TAT jobs are cancelled and new ones are scheduled for the next level:

// approval.service.ts - approveLevel()
// Cancel current level TAT jobs
await tatSchedulerService.cancelTatJobs(requestId, levelId);

// Schedule TAT jobs for next level
await tatSchedulerService.scheduleTatJobs(
  nextRequestId,
  nextLevelId,
  nextApproverId,
  nextTatHours,
  now
);

3. Rejection Flow

When a level is rejected, all pending TAT jobs are cancelled:

// approval.service.ts - approveLevel()
await tatSchedulerService.cancelTatJobs(requestId, levelId);

Notification Flow

50% TAT Alert (⏳)

Message: "50% of TAT elapsed for Request REQ-XXX: [Title]"

Actions:

• Send push notification to approver
• Update tat50_alert_sent = true
• Update tat_percentage_used = 50
• Log activity: "50% of TAT time has elapsed"

75% TAT Alert (⚠️)

Message: "75% of TAT elapsed for Request REQ-XXX: [Title]. Please take action soon."

Actions:

• Send push notification to approver
• Update tat75_alert_sent = true
• Update tat_percentage_used = 75
• Log activity: "75% of TAT time has elapsed - Escalation warning"

100% TAT Breach (⏰)

Message: "TAT breached for Request REQ-XXX: [Title]. Immediate action required!"

Actions:

• Send push notification to approver
• Update tat_breached = true
• Update tat_percentage_used = 100
• Log activity: "TAT deadline reached - Breach notification"

Configuration

Environment Variables

# Redis connection for TAT queue
REDIS_URL=redis://localhost:6379

# Optional: TAT monitoring settings
TAT_CHECK_INTERVAL_MINUTES=30
TAT_REMINDER_THRESHOLD_1=50
TAT_REMINDER_THRESHOLD_2=80

Docker Compose

Redis service is automatically configured:

redis:
  image: redis:7-alpine
  container_name: re_workflow_redis
  ports:
    - "6379:6379"
  volumes:
    - redis_data:/data
  networks:
    - re_workflow_network
  restart: unless-stopped

Working Hours Configuration

Default Schedule:

• Working Days: Monday - Friday
• Working Hours: 9:00 AM - 6:00 PM (9 hours/day)
• Timezone: Server timezone

To Modify: Edit WORK_START_HOUR and WORK_END_HOUR in tatTimeUtils.ts

Example Scenario

Scenario: 48-hour TAT Approval

1. Workflow Submitted: Monday 10:00 AM
2. 50% Alert (24 hours): Tuesday 10:00 AM
   • Notification sent to approver
   • Database updated: tat50_alert_sent = true
3. 75% Alert (36 hours): Wednesday 10:00 AM
   • Escalation warning sent
   • Database updated: tat75_alert_sent = true
4. 100% Breach (48 hours): Thursday 10:00 AM
   • Breach alert sent
   • Database updated: tat_breached = true

Error Handling

Queue Job Failures

• Automatic Retry: Failed jobs retry up to 3 times with exponential backoff
• Error Logging: All failures logged to console and logs
• Non-Blocking: TAT failures don't block workflow approval process

Redis Connection Failures

• Graceful Degradation: Application continues to work even if Redis is down
• Reconnection: Automatic reconnection attempts
• Logging: Connection status logged

Monitoring & Debugging

Check Queue Status

# View jobs in Redis
redis-cli
> KEYS bull:tatQueue:*
> LRANGE bull:tatQueue:delayed 0 -1

View Worker Logs

# Check worker status in application logs
grep "TAT Worker" logs/app.log
grep "TAT Scheduler" logs/app.log
grep "TAT Processor" logs/app.log

Database Queries

-- Check TAT status for all approval levels
SELECT 
  level_id, 
  request_id, 
  approver_name,
  tat_hours,
  tat_percentage_used,
  tat50_alert_sent,
  tat75_alert_sent,
  tat_breached,
  level_start_time,
  tat_start_time
FROM approval_levels
WHERE status IN ('PENDING', 'IN_PROGRESS');

-- Find breached TATs
SELECT * FROM approval_levels WHERE tat_breached = true;

Best Practices

1. Always Schedule on Level Start: Ensure tatStartTime is set when a level becomes active
2. Always Cancel on Level Complete: Cancel jobs when level is approved/rejected to avoid duplicate notifications
3. Use Job IDs: Unique job IDs (tat50-{requestId}-{levelId}) allow easy cancellation
4. Monitor Queue Health: Regularly check Redis and worker status
5. Test with Short TATs: Use short TAT durations in development for testing

Troubleshooting

Notifications Not Sent

1. Check Redis connection: redis-cli ping
2. Verify worker is running: Check logs for "TAT Worker: Initialized"
3. Check job scheduling: Look for "TAT jobs scheduled" logs
4. Verify VAPID configuration for push notifications

Duplicate Notifications

1. Ensure jobs are cancelled when level is completed
2. Check for duplicate job IDs in Redis
3. Verify tat50_alert_sent and tat75_alert_sent flags

Jobs Not Executing

1. Check system time (jobs use timestamps)
2. Verify working hours calculation
3. Check job delays in Redis
4. Review worker concurrency and rate limits

Future Enhancements

1. Configurable Working Hours: Allow per-organization working hours
2. Holiday Calendar: Skip public holidays in TAT calculations
3. Escalation Rules: Auto-escalate to manager on breach
4. TAT Dashboard: Real-time visualization of TAT statuses
5. Email Notifications: Add email alerts alongside push notifications
6. SMS Notifications: Critical breach alerts via SMS

API Endpoints (Future)

Potential API endpoints for TAT management:

GET  /api/tat/status/:requestId       - Get TAT status for request
GET  /api/tat/breaches                - List all breached requests
POST /api/tat/extend/:levelId         - Extend TAT for a level
GET  /api/tat/analytics                - TAT analytics and reports

References

• BullMQ Documentation
• Redis Documentation
• Day.js Documentation
• Web Push Notifications

---

Last Updated: November 4, 2025 Version: 1.0.0 Maintained By: Royal Enfield Workflow Team