Earlier today, technical issues related to SMS send-outs affected our API connections. For 30 minutes, the Voyado system could not process membership commands, both in-store and for e-commerce. During this period, SMS delivery was slower than usual. The issue was resolved, and all services have been fully operational since 10:50.
We closed the incident when the API's had been stable for some time and SMS was back at full capacity. We also advertised that we made sure that any delayed messages were to be corrected.
Current Status: We still have a small portion of recipients that haven't received their SMS message (about 5%). We have decided not to send these messages to avoid the risk of sending duplicates. During the aftermath of the incident we discovered that a central part of our sending pipeline had been affected in a way we didn't anticipate which had led to some recipients receiving multiple copies of the messages intended for them. Our top priority is resolving the core issue to ensure no further duplicate messages will be sent.
As a result of the previous disturbance with SMS messages, SMS statistics are currently not accurate and may still show some communications as sending. They will be changed to Sent or Stopped at a later point.
We are taking all possible measures to prevent this from happening again. The problem was caused by a part of our system connected to SMS send-outs that did not auto-scale as it should have, resulting in a significant backlog that couldn’t be processed quickly enough. Over time, this issue began affecting other parts of the system.
We identified the component causing the error, and once resolved, the rest of the system issues were cleared. We are now closely monitoring this to ensure stability moving forward.
We apologize to all customers affected by these technical issues. We are fully staffed with on-call teams to ensure the platform operates smoothly, and errors are handled and resolved as quickly as possible.
For further questions or concerns, please get in touch with our Support team.