Summary
On the morning of November 28, we experienced a delay in the systems responsible for processing message statistics in Voyado Engage. While message delivery worked as expected, statistics—such as opens, clicks, and bounces—were shown with a delay of up to 60 minutes in the user interface.
Customer Impact
- Message delivery was not affected.
- Users saw delays of up to 60 minutes when reviewing delivery statistics for email send-outs.
- Segmentation based on message statistics continued to work, though with a delay.
- This affected all customers using Engage to send messages during the morning and early afternoon, the impact was isolated to a delay in viewing the statistics for the messages sent.
Root Cause
The issue was caused by limitations in how our internal message handler for message statistics in particular is scaled during high load. Specifically, the service handling statistics did not automatically increase its capacity as expected due to how scaling logic was configured.
Mitigation
- Manual scaling was applied to increase processing capacity.
- Services were restarted to resolve blocks in the processing.
- We adjusted the internal message handling strategy to improve throughput.
- By mid-afternoon, statistics processing had caught up and delays were resolved.
Next Steps
- Review and update scaling configuration to ensure automatic scaling behaves as expected.
We sincerely apologize for the inconvenience this may have caused and appreciate your patience while we worked to resolve the issue.