Suyog Acharya
Case study · Message Center · Logging Module

How 20+ apps send messages and raise alarms

Every invoice, sign-up email, campaign and error alert at the company goes through two systems. I own the messaging platform end to end, and I built the logging and alerting system from scratch. Play the map below to watch a message travel.

Role
Own Message Center end to end. Built the Logging Module and its Service Manager integration.
Stack
Laravel, MySQL, AWS SES and SNS, Node.js, Express, TypeScript, MongoDB, Agenda
Scale
20+ apps, about 3.3M email logs, millions of log events
Systems
Message Center, Logging Module, Service Manager

In short: the company runs more than 20 apps. All of them need to email, text or alert people, and all of them need someone to notice when they break.

Message Center is the one place every app sends its messages through. I own how a message moves through it, from the request to the delivery report. The Logging Module is the place every app reports what happened. I built it from scratch, and when something fails it finds the responsible person and alerts them through Message Center.

The platform map

Pick a scenario and press play. Each step explains what is happening in plain words. Turn on For engineers to see the technical detail behind each step.

Message Center and the Logging Module, live

Pick a scenario

Twenty-plus apps on the left. Message Center in the middle. Email, text and app alerts on the right. The Logging Module along the bottom.

Sent in this demo Invoices 0 Billing 0 Sign-ups 0 Alerts 0 Campaigns 0 Updates 0
The structure, lane limits, worker count and retry rules match the real systems. Counts in the demo are illustrative. Scroll sideways on a small screen.

Message Center: one door for every message

Message Center is the notification backbone for more than 20 in-house apps. It carries invoices, billing notices, enrollment emails, error alerts, marketing campaigns and general updates. So far it holds about 3.3 million email logs and 4.7 million recipient rows.

I did not write its first version. I own its end-to-end flow today: how a request becomes messages, how those messages are queued and sent, and how delivery results come back. I work across all of it, from the API to the workers to the admin screens.

The rule that holds it together is simple. No app talks to an email or SMS provider directly. Every app calls Message Center. That gives the company one place for opt-outs, one place for bounce handling, one history of everything sent, and one set of limits that protects the providers.

  • Email goes through AWS SES.
  • Text messages go through AWS SNS, with Twilio as a fallback.
  • App alerts go through each app's own push API.

It is a Laravel app on MySQL. Background work runs on a database-backed queue, processed by a pool of eight supervisor-managed workers.

Lanes and the minute timer

The problem

One app might ask for a single password-reset email. Another might ask for 50,000 renewal emails in the same second. If both go in one line, the person waiting for the reset waits behind 50,000 strangers. And if all 50,000 go out at once, the email provider throttles or rejects them.

The decision

When a request arrives, a background job writes one message per recipient. The request then joins one of three lanes based on how many people it is for:

LaneRecipients
High5 or fewer
Medium6 to 49
Low50 or more

A scheduler runs every minute, once per lane. Each run releases at most 80 messages from its lane, and a run never starts while the previous one for that lane is still going. The workers then send, with up to three tries each.

Why it works

Each lane has its own timer, so the fast lane never waits behind the slow one. The cap per minute keeps the send rate steady and predictable for the providers. A huge campaign just takes longer, while everyone else's messages keep moving.

See it in the map: play Invoices go out and watch the low lane drain in capped batches.

Listening to the results

Sending is half the job. The other half is knowing what happened. SES reports delivery, opens, bounces and complaints. Those events arrive through an SNS webhook, and Message Center updates each recipient's status.

  • Five bounces on one address puts it on a block list automatically. Every later send checks that list, so the company stops paying to email a dead address and protects its sender reputation.
  • A complaint becomes an unsubscribe rule.
  • One-click unsubscribe headers go on emails, so a member can opt out from their inbox. The rule lives in Message Center, so every one of the 20+ apps respects it.
  • Status changes are pushed live to the admin screens over WebSocket, and analytics are rolled up every night.

The problem

Support staff search the email history all day to answer "did this person get their invoice?". With millions of rows, a search took about 2.96 seconds.

What I found

  • The search used a LIKE '%term%' match on a column with no index. A leading wildcard forces the database to read every row.
  • The total row count for pagination was computed twice.
  • The query loaded related records that the list screen never showed.

The decision

I removed the leading-wildcard match, removed the duplicate count and dropped the unneeded eager loads. I also added an exact-match mode that looks up by id first, which is what staff use most when they already have a reference. None of this needed a migration, so it shipped with no downtime and no risk to existing data.

2.96ssearch before
0.30ssearch after, about ten times faster

The emails that stayed "Queued" forever

In production, some email logs never left the Queued status. Nothing had crashed and no error showed up in the admin screen.

I traced it to the send job. The SMTP connection was failing with a 535 authentication error. The job caught that error itself, so the queue saw the job finish normally. But the job never marked the parent email log as failed. The log sat at Queued, and nobody could tell a slow send from a dead one.

Lesson I apply everywhere now

If code catches an error, it owns the job of recording that failure. Catching it silently turns a loud problem into an invisible one.

Logging Module: built from scratch

The problem

Each app logged in its own way, if at all. When something broke, the team found out from users. There was no single place to see errors across apps and no way to send the right person an alert.

I designed and built the Logging Module from an empty repository, in Express, TypeScript and MongoDB. Apps send it log events. It stores them, counts them, makes them searchable and raises alerts. It now holds millions of log events.

Decision 1: reject bad data at the door

A log store fills with junk fast if anything is accepted. Every event is validated strictly. The application must already be registered. The environment must be production, qa or staging. The type must be success, error, debug, info or warning. Anything else gets a 422 straight back, so the sending app finds out at once.

Decision 2: split by type, index by how people search

Events go into three collections: general logs, success logs and error logs. Errors are what people look at most, so they never have to be filtered out of a pile of successes. Each collection has compound indexes that match the real queries:

IndexServes
{application, createdAt}"Show me this app's recent logs"
{createdAt, environment}"Everything in production today"
{environment, type, createdAt}"Production errors this week"

When someone wants one list across all three, a single aggregation with $unionWith merges and paginates them in the database.

Decision 3: count on write, so dashboards never scan

A dashboard that counts raw logs gets slower every day the system runs. Instead, every time an event is saved, the module bumps an hourly and a daily counter for that app, environment and type. It uses an $inc upsert into a statistics collection and does not wait for it. The dashboard reads a few hundred small counter documents instead of millions of logs.

Decision 4: search that ranks what matters

Search uses a weighted text index: feature name counts most (10), then tags (8), then detail (5), then environment (3). Popular searches are cached in memory for 60 seconds. Adding search to existing data needed a new field on old documents, so I wrote a migration that walks the collection with a cursor and writes in batches of 500 with bulkWrite, which keeps memory flat on millions of rows.

Decision 5: alert a person, not a mailbox

An alert rule says who is responsible: one person or a notification group, by email, text or both. When a matching error arrives, the module resolves the recipients and calls Message Center for email and text in parallel. The Logging Module is just another client of Message Center, so alerts get the same fast lane, retries and history as any other message.

Access and tests

Access is role-based: 19 permissions grouped into 5 roles (Super Admin, Admin, Manager, Developer, Viewer), checked on every request against the user's JWT. The module has about 1,100 lines of Mocha, Chai and Sinon tests running against an in-memory MongoDB, so tests are fast and never touch a shared database.

Service Manager: logging without making anyone wait

The problem

Service Manager needed to log every API call it handles. The simple way is to send the log before answering. That adds a network round trip to every request, and if the Logging Module is slow or down, Service Manager becomes slow or down too.

The decision

I built the integration so responses never wait for logging.

  1. For every API call, a logging service builds one record: request, response, method, url and duration. A status of 400 or above marks it as an error.
  2. It hands the record to a job queue (Agenda) and moves on. It does not wait for the hand-off to finish.
  3. A separate worker process, run by pm2, picks up jobs every second, up to 10 at once overall and 5 for this job type. It posts each record to the Logging Module.
  4. On success the job is removed. On failure it is retried up to three times, an hour apart, and the reason is saved on the job.
  5. The worker shuts down cleanly on SIGTERM and SIGINT, so a deploy never drops a job halfway.

I also moved Agenda's job collection out of the logging database, so queue churn never competes with log storage.

Why it matters

If the Logging Module goes down for an hour, Service Manager users notice nothing. The logs wait in the queue and arrive when it comes back.

Play Something breaks in the map to watch this path end to end, from the failed request to a text on the team's phones.

Results

20+apps sending through one platform with shared opt-outs, limits and history
2.96s → 0.30semail-log search, with no migration
Millionslog events in a system I built from scratch

Apps hand off a message or a log and move on. Big sends never block urgent ones. Dead addresses block themselves. When something breaks, a named person hears about it in minutes, and the dashboard shows it without scanning raw data.

What I would do next

Replace the fixed one-hour retry gap with a growing backoff, add an alert when a lane's backlog grows faster than it drains, and record the final failure reason on every message so stuck statuses can never come back.