Engage: performance issues affecting tracking pipeline (abandoned cart/products of interests) and reports

Summary

Between June 4 at 14:39 and June 5 at 14.10, one of our backend services experienced performance issues leading to delays and partial unavailability. The issue has been resolved, and systems are now operating normally.

Customer Impact

During the incident, features such as ROD, Abandoned Cart, Register Status, Automation Overview and Product of interest were affected. Customers may have experienced slow response times or temporary unavailability when using these functionalities. No data loss occurred, and all services were restored by June 5 at 14.10.

Root Cause and Mitigation

The issue stemmed from an unexpected delay in the cleanup of background processes connected to secure storage systems, following a minor system update. While the connections themselves were successful, the delayed termination caused performance degradation. We mitigated the issue by safely bypassing the affected cleanup process and restored service stability. A support case was also raised with Microsoft Azure to investigate further.

Next Steps

• Improve automated testing to detect performance issues before deployment.
• Review and update the logic used for secure storage connections in production environments.
• Monitor Azure infrastructure updates for changes that may affect our services.
• Continue to enhance logging and observability for faster diagnostics.