Incident Summary

On the 17th January 2024, we deployed a new feature for the Voyado Engage app, allowing customers to monitor the application's health and receive alerts in case of errors. Unfortunately, an issue arose within the Monitor service, leading to the distribution of a considerable number of alerts containing false negative information. These alerts were sent to all Shopify Store Owners who have the app installed.

Impact

The incident resulted in a surge of misleading alerts, causing confusion and potential concern among our customer base.

Root Cause

The root cause of the incident was identified as a flaw in the Monitor service logic, leading to the generation and communication of alerts based on inaccurate information. This flaw resulted in an unintended and widespread communication of false negatives.

Resolution

As of now, most errors have been resolved.. However, we acknowledge that there are still lingering false negative errors specifically tied to the Refund Sync and the Shopify POS Order sync features. Our team is actively working to address these remaining issues promptly.

Mitigation Steps Taken

1. Immediate attention was given to rectify the flawed logic within the Monitor service.
2. Continuous monitoring and testing are in place to ensure that similar issues do not recur in the future.

Next Steps

1. Complete the resolution of the remaining false negative errors related to the Refund Sync and the Shopify POS Order sync features.
2. Conduct a thorough review of our release and monitoring processes to prevent similar incidents in future updates.
3. Enhance communication channels to keep our users informed about incident resolution progress and preventive measures.

Conclusion

We sincerely apologize for any inconvenience caused by this incident. If you have any further questions or concerns, please do not hesitate to reach out to our support team (support@voyado.com).