Re_Backend/docs/TAT_TESTING_GUIDE.md

TAT Notification Testing Guide

Quick Setup for Testing

Step 1: Setup Redis

You MUST have Redis for TAT notifications to work.

🚀 Option A: Upstash (RECOMMENDED - No Installation!)

Best choice for Windows development:

1. Go to: https://console.upstash.com/
2. Sign up (free)
3. Create Database:
   • Name: redis-tat-dev
   • Type: Regional
   • Region: Choose closest
4. Copy Redis URL (format: rediss://default:...@host.upstash.io:6379)
5. Add to Re_Backend/.env:

   REDIS_URL=rediss://default:YOUR_PASSWORD@YOUR_HOST.upstash.io:6379
   

✅ Done! No installation, works everywhere!

See detailed guide: docs/UPSTASH_SETUP_GUIDE.md

Option B: Docker (If you prefer local)

docker run -d --name redis-tat -p 6379:6379 redis:latest

Then in .env:

REDIS_URL=redis://localhost:6379

Option C: Linux Production

sudo apt install redis-server -y
sudo systemctl start redis-server

Verify Connection

• Upstash: Use Console CLI → PING → should return PONG
• Local: Test-NetConnection localhost -Port 6379

---

Step 2: Enable Test Mode (Optional but Recommended)

For faster testing, enable test mode where 1 hour = 1 minute:

1. Edit your .env file:

TAT_TEST_MODE=true

2. Restart your backend:

cd Re_Backend
npm run dev

3. Verify test mode is enabled - You should see:

⏰ TAT Configuration:
   - Test Mode: ENABLED (1 hour = 1 minute)
   - Working Hours: 9:00 - 18:00
   - Working Days: Monday - Friday
   - Redis: redis://localhost:6379

---

Step 3: Create a Test Workflow

Production Mode (TAT_TEST_MODE=false)

• Create a request with 2 hours TAT
• Notifications will come at:
  • 1 hour (50%)
  • 1.5 hours (75%)
  • 2 hours (100% breach)

Test Mode (TAT_TEST_MODE=true) ⚡ FASTER

• Create a request with 6 hours TAT (becomes 6 minutes)
• Notifications will come at:
  • 3 minutes (50%)
  • 4.5 minutes (75%)
  • 6 minutes (100% breach)

---

Step 4: Submit and Monitor

1. Create and Submit Request via your frontend or API

2. Check Backend Logs - You should see:

[TAT Scheduler] Calculating TAT milestones for request...
[TAT Scheduler] Start: 2025-11-04 12:00
[TAT Scheduler] 50%: 2025-11-04 12:03
[TAT Scheduler] 75%: 2025-11-04 12:04
[TAT Scheduler] 100%: 2025-11-04 12:06
[TAT Scheduler] Scheduled tat50 for level...
[TAT Scheduler] Scheduled tat75 for level...
[TAT Scheduler] Scheduled tatBreach for level...
[TAT Scheduler] ✅ TAT jobs scheduled for request...

3. Wait for Notifications

   • Watch the logs
   • Check push notifications
   • Verify database updates

4. Verify Notifications - Look for:

[TAT Processor] Processing tat50 for request...
[TAT Processor] tat50 notification sent for request...

---

Testing Scenarios

Scenario 1: Normal Flow (Happy Path)

1. Create request with TAT = 6 hours (6 min in test mode)
2. Submit request
3. Wait for 50% notification (3 min)
4. Wait for 75% notification (4.5 min)
5. Wait for 100% breach (6 min)

Expected Result:

• ✅ 3 notifications sent
• ✅ Database flags updated
• ✅ Activity logs created

---

Scenario 2: Early Approval

1. Create request with TAT = 6 hours
2. Submit request
3. Wait for 50% notification (3 min)
4. Approve immediately
5. Remaining notifications should be cancelled

Expected Result:

• ✅ 50% notification received
• ✅ 75% and 100% notifications cancelled
• ✅ TAT jobs for next level scheduled

---

Scenario 3: Multi-Level Approval

1. Create request with 3 approval levels (2 hours each)
2. Submit request
3. Level 1: Wait for notifications, then approve
4. Level 2: Should schedule new TAT jobs
5. Level 2: Wait for notifications, then approve
6. Level 3: Should schedule new TAT jobs

Expected Result:

• ✅ Each level gets its own TAT monitoring
• ✅ Previous level jobs cancelled on approval
• ✅ New level jobs scheduled

---

Scenario 4: Rejection

1. Create request with TAT = 6 hours
2. Submit request
3. Wait for 50% notification
4. Reject the request
5. All remaining notifications should be cancelled

Expected Result:

• ✅ TAT jobs cancelled
• ✅ No further notifications

---

Verification Checklist

Backend Logs ✅

# Should see these messages:
✓ [TAT Queue] Connected to Redis
✓ [TAT Worker] Initialized and listening
✓ [TAT Scheduler] TAT jobs scheduled
✓ [TAT Processor] Processing tat50
✓ [TAT Processor] tat50 notification sent

Database Check ✅

-- Check approval level TAT status
SELECT 
  request_id,
  level_number,
  approver_name,
  tat_hours,
  tat_percentage_used,
  tat50_alert_sent,
  tat75_alert_sent,
  tat_breached,
  tat_start_time,
  status
FROM approval_levels
WHERE request_id = '<YOUR_REQUEST_ID>';

Expected Fields:

• tat_start_time: Should be set when level starts
• tat50_alert_sent: true after 50% notification
• tat75_alert_sent: true after 75% notification
• tat_breached: true after 100% notification
• tat_percentage_used: 50, 75, or 100

Activity Logs ✅

-- Check activity timeline
SELECT 
  activity_type,
  activity_description,
  user_name,
  created_at
FROM activities
WHERE request_id = '<YOUR_REQUEST_ID>'
ORDER BY created_at DESC;

Expected Entries:

• "50% of TAT time has elapsed"
• "75% of TAT time has elapsed - Escalation warning"
• "TAT deadline reached - Breach notification"

Redis Queue ✅

# Connect to Redis
redis-cli

# Check scheduled jobs
KEYS bull:tatQueue:*
LRANGE bull:tatQueue:delayed 0 -1

# Check job details
HGETALL bull:tatQueue:tat50-<REQUEST_ID>-<LEVEL_ID>

---

Troubleshooting

❌ No Notifications Received

Problem: TAT jobs scheduled but no notifications

Solutions:

1. Check Redis is running:

   Test-NetConnection localhost -Port 6379
   

2. Check worker is running:

   # Look for in backend logs:
   [TAT Worker] Worker is ready and listening
   

3. Check job delays:

   redis-cli
   > LRANGE bull:tatQueue:delayed 0 -1
   

4. Verify VAPID keys for push notifications:

   # In .env file:
   VAPID_PUBLIC_KEY=...
   VAPID_PRIVATE_KEY=...
   

---

❌ Jobs Not Executing

Problem: Jobs scheduled but never execute

Solutions:

1. Check system time is correct
2. Verify test mode settings
3. Check worker logs for errors
4. Restart worker:

   # Restart backend server
   npm run dev
   

---

❌ Duplicate Notifications

Problem: Receiving multiple notifications for same milestone

Solutions:

1. Check database flags are being set:

   SELECT tat50_alert_sent, tat75_alert_sent FROM approval_levels;
   

2. Verify job cancellation on approval:

   # Should see in logs:
   [Approval] TAT jobs cancelled for level...
   

3. Check for duplicate job IDs in Redis

---

❌ Redis Connection Errors

Problem: ECONNREFUSED errors

Solutions:

1. Start Redis - See Step 1
2. Check Redis URL in .env:

   REDIS_URL=redis://localhost:6379
   

3. Verify port 6379 is not blocked:

   Test-NetConnection localhost -Port 6379
   

---

Testing Timeline Examples

Test Mode Enabled (1 hour = 1 minute)

TAT Hours	Real Time	50%	75%	100%
2 hours	2 minutes	1m	1.5m	2m
6 hours	6 minutes	3m	4.5m	6m
24 hours	24 minutes	12m	18m	24m
48 hours	48 minutes	24m	36m	48m

Production Mode (Normal)

TAT Hours	50%	75%	100%
2 hours	1h	1.5h	2h
6 hours	3h	4.5h	6h
24 hours	12h	18h	24h
48 hours	24h	36h	48h

---

Quick Test Commands

# 1. Check Redis
Test-NetConnection localhost -Port 6379

# 2. Start Backend (with test mode)
cd Re_Backend
$env:TAT_TEST_MODE="true"
npm run dev

# 3. Monitor Logs (in another terminal)
cd Re_Backend
Get-Content -Path "logs/app.log" -Wait -Tail 50

# 4. Check Redis Jobs
redis-cli KEYS "bull:tatQueue:*"

# 5. Query Database
psql -U laxman -d re_workflow_db -c "SELECT * FROM approval_levels WHERE tat_start_time IS NOT NULL;"

---

Support

If you encounter issues:

1. Check Logs: Re_Backend/logs/
2. Enable Debug: Set LOG_LEVEL=debug in .env
3. Redis Status: redis-cli ping should return PONG
4. Worker Status: Look for "TAT Worker: Initialized" in logs
5. Database: Verify TAT fields exist in approval_levels table

---

Happy Testing! 🎉

For more information, see:

• TAT_NOTIFICATION_SYSTEM.md - Full system documentation
• INSTALL_REDIS.txt - Redis installation guide
• backend_structure.txt - Database schema reference