[Engage] API issues

Incident Report for Voyado

Postmortem

Summary

On April 30th 2025, between 15:49 and 16:38 CEST, Voyado Engage experienced an issue that caused delays and failures in API calls. The issue was traced to a problem during a system deployment. Our engineering team identified and resolved the issue quickly, with full service restored by 17:03.

Customer Impact

During the incident, a significant number of API calls were delayed or failed. Even successful calls had noticeably slower response times.

Root Cause and Mitigation

The issue was caused by a misconfiguration in a system update. This affected how the platform processed internal requests and resulted in service degradation. When the deployment was paused to limit impact, some servers were left running the old version of Engage and could not handle the full traffic load.

Our team halted the rollout, developed a fix, and redeployed the updated version. Recovery began at 16:38, with full functionality confirmed by 17:03.

Next Steps

To prevent similar issues in the future, we are taking actions to implement improvements to our deployment and monitoring processes and to ensure faster recovery time.

We appreciate your patience and understanding, and apologize for any inconvenience. We remain committed to providing a stable and reliable platform experience.

Posted May 13, 2025 - 12:07 CEST

Resolved

This incident has been resolved.
Posted Apr 30, 2025 - 17:41 CEST

Monitoring

We’re continuing to see improvements and are getting back to normal operations. We're now monitoring the fix closely.
Posted Apr 30, 2025 - 17:08 CEST

Update

The fix is currently being rolled out, and we're starting to see signs of improvement. However, the issue is not fully resolved yet.
Posted Apr 30, 2025 - 16:50 CEST

Identified

We have identified the issue and are working on a fix.
Posted Apr 30, 2025 - 16:20 CEST

Investigating

We are currently investigating a possible incident affecting our APIs.
Posted Apr 30, 2025 - 16:09 CEST
This incident affected: Engage (API).