Sahil Ashar

Scalable Notification System

shipped Impact: 9/10
#infrastructure #golang #microservices

Overview

Designed and implemented a scalable notification system to consolidate email, SMS, and push notifications across all company products.

Problem

The company had 4 different notification systems scattered across the codebase:

  • Each team maintained their own email templates
  • No unified delivery tracking
  • Poor retry logic led to dropped notifications
  • No rate limiting caused spam complaints

Solution

Built a centralized notification service with:

Architecture

  • Go microservice with gRPC API
  • PostgreSQL for delivery tracking
  • Redis for rate limiting and deduplication
  • RabbitMQ for async processing
  • Template engine for multi-channel rendering

Key Features

  • Unified API for all notification types
  • Smart retry with exponential backoff
  • User preference management
  • Delivery analytics dashboard
  • A/B testing framework for content

Impact

  • 5M+ notifications sent daily
  • 99.9% delivery success rate
  • 60% reduction in notification-related bugs
  • $50K/year saved in third-party costs
  • 2 engineers freed from maintenance work

Technical Challenges

Challenge 1: Rate Limiting

Implemented token bucket algorithm per user and per channel to respect rate limits while maximizing throughput.

Challenge 2: Delivery Guarantees

Designed idempotent delivery system with at-least-once semantics and client-side deduplication.

Challenge 3: Template Management

Built a custom DSL for templates that works across email, SMS, and push with automatic variable validation.

Results

The system now handles 100% of company notifications and has become the foundation for our customer engagement platform.

Tech Stack

  • Backend: Go, gRPC, PostgreSQL, Redis, RabbitMQ
  • Infrastructure: Kubernetes, AWS (SQS, SNS, SES)
  • Monitoring: Prometheus, Grafana, PagerDuty