Email Recommendation Out of Memory

Due to a recent change by a customer, where they started using very large images, a memory leak in Email Recommendations was exposed and led to the application crashing.

The memory leak has now been fixed, and a permanent solution for the customer has been applied to reduce the size of the images created by the application.

• We will implement a feature in Email Recommendations that will give an error if the images are too big.

  • Documentation will be updated with more information about this.

• Implemented a self-healing mechanism in case this issue should appear again.

The incident also exposed an internal problem with a service providing a backend application with data that did not recover automatically after an incident. This issue has been resolved, and the affected service will now recover automatically.

The business impact was that the service was unavailable for:

• 1 hour 45 minutes for end users
• 2 hours 15 minutes for backend users + 1 hour for the customer with large images.

We apologize for any inconvenience this has caused and will work to ensure it won't happen again.