[Engage] Report Register status and Automation statistics are not working

Incident Report for Voyado

Postmortem

Summary
On March 11th, two different issues were identified that impacted report functionality within the Engage platform. The first issue affected the Register Status Report by preventing desired report results and the second issue prevented data publication for most tenants in the Automation Statistics Report.

Customer Impact
Customers experienced difficulties accessing accurate reporting data for about a (1) day before it was functioning as expected the following day. The respective report was impacted:

  • Register Status Report
  • Automation Statistics Report

Root Cause

  • Register Status Report Issue: A change in how a data type were handled during refactoring led to an oversight in compatibility with an API regarding null values.
  • Automation Statistics Report Issue: Missing necessary tables to be utilized for the report in the production environment data layer. A scheduled job to create these tables was not executed, leading to a mismatch between Engage logic and the Data layer API.

Remediation
Immediate actions were taken to resolve the issues:

  • Register Status Report: A hotfix was deployed to fix the bug.
  • Automation Statistics Report: The necessary job was manually triggered to create the missing tables, restoring the report functionality.

Next Steps
To prevent similar incidents in the future, we have identified improvements to be implemented in areas of testing and validation and to continue to follow established processes.

We appreciate your patience and remain committed to delivering a seamless and high-quality platform experience. If you have any further questions or concerns, please reach out to our support team.

Posted Mar 20, 2025 - 15:31 CET

Resolved

Both the Automation statistics report and Register status is now working again.
Posted Mar 11, 2025 - 13:25 CET

Update

Register status is now working. We are still working on a fix for the Automation statistics report.
Posted Mar 11, 2025 - 11:43 CET

Identified

We have identified issues with our reports Register status and Automation statistics. They are currently not loading any data. We are working on a fix for this.
Posted Mar 11, 2025 - 10:18 CET
This incident affected: Engage (Other).