[Engage] v3 Orders Currently Unavailable

Incident Report for Voyado

Postmortem

Summary

On October 1st, 2025, the Order V3 API was unavailable between 12:00 and 14:30 CEST. The issue was caused by a faulty update intended to handle previously failed orders and led to serialization errors that made the API unresponsive. The problem was identified and a fix was deployed to restore normal operations.

Customer Impact

A few customers were affected during the incident. Customers using the Order V3 API were unable to place or manage orders through this version of the API during the downtime. Other versions and platform features remained unaffected.

Root Cause

The issue derived from a serialization error introduced during a fix deployment aimed at resending failed order messages. This error caused all incoming commands to fail, rendering the Order V3 API temporarily unusable.

Mitigation

The issue was identified and a fix was prepared and deployed. Full functionality was restored by 14:30 CEST, the system was closely monitored to ensure stability.

Next Steps

We are taking several actions to prevent similar incidents:

  • Improving test coverage and automated validation for critical API updates.
  • Enhancing our developer tools to better simulate and detect serialization issues.

We apologize for the inconvenience caused and appreciate your patience as we continue to improve the reliability of our platform.

Posted Oct 15, 2025 - 11:40 CEST

Resolved

This incident has been resolved.
Posted Oct 01, 2025 - 15:02 CEST

Monitoring

The hotfix deployment has been successfully released, and services are back to normal operation.
Posted Oct 01, 2025 - 14:23 CEST

Update

The hotfix deployment is still ongoing.
Posted Oct 01, 2025 - 14:06 CEST

Update

A hotfix is currently being deployed. The estimated time until the Orders v3 endpoint is available again is approximately 60 minutes.
Posted Oct 01, 2025 - 12:45 CEST

Identified

We are currently experiencing a disruption affecting customers using the Orders v3 endpoint. A fix is in progress and will be deployed shortly.
Posted Oct 01, 2025 - 12:27 CEST
This incident affected: Engage (API).