Summary

On September 30th, Engage experienced a temporary disruption in email send-outs due to an error in newly deployed code. The issue was quickly identified, and a hotfix was deployed. While no messages were lost, some email messages were delayed as the system worked through a backlog.

Customer Impact

Majority of customers were potentially affected. Most experienced delays in automation emails. In a few cases, delays were up to 8 hours, but the majority of impacted messages were delivered within 3–4 hours. No SMS or priority messages were affected.

Root Cause

The disruption was caused by an internal misconfiguration in how the platform determined where to store and retrieve data for email messages. When a new code deployment corrected this configuration, it unintentionally disrupted messages that had already been processed using the old setup. This led to messages being queued and not sent.

Mitigation

After identifying the issue, a hotfix was deployed to prevent further impact. Additional work was done throughout the day to manually resend affected messages and ensure all queued messages were successfully delivered.

Next Steps

To reduce the risk of similar issues in the future, we are:

• Implementing automated code checks that will detect and block misconfigurations before deployment.
• Improving internal processes for incident handling and maintenance tooling, including procedures to quickly resending impacted messages.

We apologize for any inconvenience caused and appreciate your patience while we resolved the issue.