Voyado Status

Dec 11, 2024

Completed - The scheduled maintenance has been completed.
Dec 11, 00:00 CET

In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
Dec 10, 21:00 CET

Scheduled - Every Tuesday at 21:00 CET/CEST we reserve the right to perform scheduled maintenance on Voyado Engage. Normally this won't affect availability at all, but on some rare occasions unresponsiveness or degraded performance may be experienced during the upgrade.

We do apologize in advance for any impact on your work and/or availability, but hope that the things we release to production will make it all worth it in the end.
Sep 2, 14:46 CEST

Dec 10, 2024

Dec 9, 2024

No incidents reported.

Dec 8, 2024

Voyado Engage - Service window

Completed - The scheduled maintenance has been completed.
Dec 8, 05:00 CET

In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
Dec 8, 03:00 CET

Scheduled - Every Sunday at 03:00 CET/CEST we reserve the right to perform scheduled maintenance on our databases. During this time we make sure that the databases are fine tuned and all your data is protected. Normally this won’t affect availability, but some unresponsiveness or degraded performance may be experienced for short periods during the maintenance.
Sep 9, 15:22 CEST

Dec 7, 2024

No incidents reported.

Dec 6, 2024

No incidents reported.

Dec 5, 2024

No incidents reported.

Dec 4, 2024

Voyado Engage- Service window

Completed - The scheduled maintenance has been completed.
Dec 4, 00:00 CET

In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
Dec 3, 21:00 CET

Scheduled - Every Tuesday at 21:00 CET/CEST we reserve the right to perform scheduled maintenance on Voyado Engage. Normally this won't affect availability at all, but on some rare occasions unresponsiveness or degraded performance may be experienced during the upgrade.

We do apologize in advance for any impact on your work and/or availability, but hope that the things we release to production will make it all worth it in the end.
Sep 2, 14:45 CEST

Dec 3, 2024

Engage- slowness in automations

Resolved - We have resolved the issue and will continue to work on the root cause.
Dec 3, 20:51 CET

Update - We are continuing to monitor for any further issues.
Dec 3, 18:37 CET

Update - We are continuing to monitor for any further issues.
Dec 3, 18:34 CET

Monitoring - We are back at normal performance for automations and will continue to monitor closely.
Dec 3, 18:30 CET

Update - We are still working to remediate the issue, status on performance is unchanged.
Dec 3, 17:44 CET

Identified - We have identified the issue and are currently in progress of verifying if our actions have resolved the issue with degraded performance on automations.
Dec 3, 17:06 CET

Investigating - We are currently experiencing larger than usual queues in our automations. We are investigating this.
Dec 3, 16:31 CET

Dec 2, 2024

No incidents reported.

Dec 1, 2024

Voyado Engage - Service window

Completed - The scheduled maintenance has been completed.
Dec 1, 05:00 CET

In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
Dec 1, 03:00 CET

Scheduled - Every Sunday at 03:00 CET/CEST we reserve the right to perform scheduled maintenance on our databases. During this time we make sure that the databases are fine tuned and all your data is protected. Normally this won’t affect availability, but some unresponsiveness or degraded performance may be experienced for short periods during the maintenance.
Sep 9, 15:22 CEST

Nov 30, 2024

[ENGAGE] INFO: SMS degradation from earlier today

Resolved - This incident has been resolved.
Nov 30, 11:19 CET

Identified - Earlier today, technical issues related to SMS send-outs affected our API connections.
For 30 minutes, the Voyado system could not process membership commands, both in-store and for e-commerce. During this period, SMS delivery was slower than usual. The issue was resolved, and all services have been fully operational since 10:50.

We closed the incident when the API's had been stable for some time and SMS was back at full capacity.
We also advertised that we made sure that any delayed messages were to be corrected.

Current Status:
We still have a small portion of recipients that haven't received their SMS message (about 5%). We have decided not to send these messages to avoid the risk of sending duplicates. During the aftermath of the incident we discovered that a central part of our sending pipeline had been affected in a way we didn't anticipate which had led to some recipients receiving multiple copies of the messages intended for them. Our top priority is resolving the core issue to ensure no further duplicate messages will be sent.

As a result of the previous disturbance with SMS messages, SMS statistics are currently not accurate and may still show some communications as sending. They will be changed to Sent or Stopped at a later point.

We are taking all possible measures to prevent this from happening again.
The problem was caused by a part of our system connected to SMS send-outs that did not auto-scale as it should have, resulting in a significant backlog that couldn’t be processed quickly enough. Over time, this issue began affecting other parts of the system.

We identified the component causing the error, and once resolved, the rest of the system issues were cleared. We are now closely monitoring this to ensure stability moving forward.

We apologize to all customers affected by these technical issues. We are fully staffed with on-call teams to ensure the platform operates smoothly, and errors are handled and resolved as quickly as possible.

For further questions or concerns, please get in touch with our Support team.
Nov 29, 16:11 CET

Nov 29, 2024

[Engage] Service degradation

Postmortem - Read details
Dec 6, 22:42 CET

Resolved - Our API:s are still stable and back at normal operations.
Sending of SMS is also back to a more normal status and we are sending at full capacity.
Teams are making sure that any delayed messages are delivered to their intended recipients, albeit somewhat late in some cases.

We sincerely apologize for the inconvenience caused by this service degradation.
We are fully staffed and working hard to monitor the platform and taking swift action to mediate any problems caused by the Black Friday load. This day is as important to us as it is to you.
Nov 29, 11:55 CET

Update - We are seeing signs of improvement regarding the API:s since approximately 10:51 CET.
Sending of SMS still degraded.
Our efforts continue at full force.
Nov 29, 10:58 CET

Update - We are continuing to investigate the issue, using all resources available to us.
Monitoring shows that our APIs are affected as well as the sending of SMS messages.

SMS are being sent, but at a lower rate than normal.
Nov 29, 10:50 CET

Investigating - We are currently investigating service degradation on our APIs
More information will follow
Nov 29, 10:34 CET

Nov 28, 2024

Voyado Engage - Emails delayed

Postmortem - Read details
Dec 4, 09:07 CET

Resolved - All the delay messages has now been sent. The incident has now been resolved.
Nov 28, 23:51 CET

Update - We still see a good effect of the implemented fix and the sendouts are going out as planned now. We are still processing the queue of the previously delayed messages.
Nov 28, 20:45 CET

Monitoring - We see good effect of a fix we implemented. Engage is coming back to normal performance. Sendouts are going out as planned now, and queues are being processed. We will monitor and update regarding send queue status.
Nov 28, 19:44 CET

Update - We are still seeing delays because of high load. Email sendouts are being queued. We are working on different solutions to make sure emails are going out. At the moment no other parts of Engage is affected.
Nov 28, 19:06 CET

Update - We are still seeing delays because of high load. Email sendouts are being queued. We are working on different solutions to make sure emails are going out. At the moment no other parts of Engage is affected.
Nov 28, 18:39 CET

Update - We are still seeing delays because of high load. Email sendouts are being queued. We are working on different solutions to make sure emails are going out. At the moment no other parts of Engage is affected.
Nov 28, 18:07 CET

Update - We are still working to find a solution. The current email sendout delay is approximately 1,5h.
Nov 28, 17:42 CET

Update - We’re still seeing delays and are working to find a solution. At the moment no other parts of Engage is affected.
Nov 28, 17:33 CET

Update - We’re still seeing delays and are working to find a solution. At the moment no other parts of Engage is affected.
Nov 28, 17:09 CET

Investigating - We are currently experiencing delays in sending emails. We are working to resolve the issue.
Nov 28, 16:34 CET

Nov 27, 2024

Voyado Engage - Emails delayed

Resolved - The incident has now been resolved, all the messages that was delayed has been resent.
Nov 27, 18:53 CET

Investigating - We are currently experiencing delays in sending emails. We are working to resolve the issue.
Nov 27, 18:01 CET

Voyado Engage- Service window

Completed - The scheduled maintenance has been completed.
Nov 27, 00:00 CET

In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
Nov 26, 21:00 CET

Scheduled - Every Tuesday at 21:00 CET/CEST we reserve the right to perform scheduled maintenance on Voyado Engage. Normally this won't affect availability at all, but on some rare occasions unresponsiveness or degraded performance may be experienced during the upgrade.

We do apologize in advance for any impact on your work and/or availability, but hope that the things we release to production will make it all worth it in the end.
Sep 2, 14:44 CEST

Related