02 — WebApp
← back

project_02

Velox

notification dispatch engine — event-driven, multi-channel, 500k msg/day

● ProductionKafka / EventsGo Workers
Overview

Velox completely decouples notification triggering from delivery. Any upstream service drops a Kafka event without knowing which channels are downstream. A central router fans out to isolated Go workers — one per channel. Each worker owns its retry logic, rate limiting, and provider failover independently, so a Twilio outage never touches SendGrid throughput.

Problem

Monolithic notification pipelines become single points of failure. One slow SMS provider stalls email delivery for unrelated users. Notification logic bleeds into business services and a single security patch requires 6 coordinated deploys across the stack.

Engineering Challenges
01.Provider failover — circuit breaker trips after 3 consecutive 5xx responses, rerouting to backup provider
02.Exactly-once delivery using idempotency tokens in Redis with 24h TTL to safely deduplicate Kafka replays
03.Per-user per-channel rate limiting without blocking sibling users in the same Kafka partition
Key Decisions
→Kafka over RabbitMQ — log retention enables full replay of missed notifications during any channel outage
→Go workers per channel — goroutines handle the concurrency profile far cheaper than Node.js thread pools
→Dead letter queue after 3 retries — no silent failures, everything is inspectable and replayable
→Redis preference store — O(1) channel lookup before every dispatch, no DB touch at 500k msg/day
← prevSunakkusourcenext →Gatekeeper
Stack
Node.jsKafkaGoMongoDBRedisSendGridTwilioFCMKubernetes
Architecture
Upstream Servicesany serviceEvent RouterKafka · Node.jsEmailSendGrid · GoSMSTwilio · GoPushFCM / APNsemit eventfan-out
Key Features
◆Channel isolation — Each delivery channel runs in its own Go worker — Twilio outage never affects SendGrid or FCM
◆Event replay — Kafka log retention enables full replay of missed notifications during outages with zero data loss
◆Circuit breaker — Automatic provider failover after 3 consecutive failures — switches to backup within milliseconds
◆Preference routing — Per-user channel preferences stored in Redis for O(1) lookup — no DB touch per dispatch at scale
Metrics
500k+Msgs / Day
99.97%Delivery Rate
<800msP99 Latency
3Channels