On November 7th, an unusually high number of internal events were generated due to a planned feature rollout. This caused delays in processing certain automations.
Automations triggered by customer interactions or audience targeting were delayed by up to 12 hours. All affected automations were eventually executed. Other parts of the platform operated as normal.
When the new Marketing Groups feature was enabled for all customers, a large volume of events (hundreds of millions) was generated in a short time. The system processing these events could not keep up, which resulted in delays for other events in the same internal queue.
To restore normal flow, we temporarily cleared out the backlog of non-critical events and performed database synchronizations to ensure data accuracy. Normal operation resumed later the same day.
We are improving our monitoring and alerting to detect similar issues earlier, and we are reviewing our event handling capacity and release procedures to prevent future impact from large-scale rollouts.
We apologize for any inconvenience caused and appreciate your patience while we resolved the issue.