[Engage] Delay in Processing of Sendouts

Summary

On September 28th, Engage experienced delays in sending bulk email messages. The issue was caused by part of our internal email processing setup not operating as expected. The delay was resolved the same day, and no messages were lost.

Customer Impact

Bulk emails for a minor sub-set of customers were delayed by up to 3 hours. Priority messages, including automation emails and SMS, were not affected and were delivered as normal.

Root Cause

Three of the internal services responsible for processing emails were not active due to a system update on September 27th. This caused them to restart without automatically reactivating the needed components. Combined with a high traffic in the system, this led to a backlog in the email messaging pipeline.

Mitigation

The inactive services were redeployed, restoring full processing capacity. Once this was done, the message queue was quickly processed, and operations returned to normal.

Next Steps

To prevent similar issues in the future, we are:

• Updating our deployment pipeline to ensure the required components are automatically restarted after updates.
• Introducing additional monitoring and alerts to detect inactive or underperforming internal services sooner.

We appreciate your understanding and patience while we worked through this issue and apologize for any inconvenience.