Postmortem -
Read details
Feb 6, 13:31 CET
Resolved -
The incident has now been resolved after active monitoring of the system after our recent fix. We have managed the delayed messages, where the majority have been resent and in a few minor cases been rescheduled.
Jan 29, 22:17 CET
Monitoring -
The fix implemented in our recent deploy has show desired effects and we are experiencing new messages (email and sms) being sent as expected. However, we have an accumulated queue of messages that should have been sent throughout the day that will experience a delay. We are working on managing the delayed messages.
Jan 29, 20:41 CET
Update -
The status of the delays remains unchanged. We are actively investigating the issue through troubleshooting and are awaiting the resolution of our recent update.
Jan 29, 20:16 CET
Update -
We are rolling out an update that includes fixes to help address the issue and additional monitoring tools to better understand the root cause.
Jan 29, 18:12 CET
Update -
We are still in active troubleshooting and delays are still present. Currently, we have an average delay on 45 min on all messages going out.
Jan 29, 16:47 CET
Update -
We are still in active troubleshooting and delays are unchanged.
Jan 29, 15:50 CET
Update -
Our attempt on mitigating the issue with implemented fix did not have the desired effect and we are continuing our troubleshooting. Delays in messaging are unchanged.
Jan 29, 15:08 CET
Update -
We are still seeing a delay after the implementing an additional fix. Our efforts to mitigate and resolve the issue continues.
Jan 29, 14:27 CET
Update -
We have identified and implemented a fix to mitigate the issue, and are seeing some improvement to the delay, however there is still a delay for messages to be sent. Troubleshooting is still ongoing to ensure our mitigation efforts are correct and to identify the root cause.
Jan 29, 13:01 CET
Update -
We are continuing to investigate this issue.
Jan 29, 12:26 CET
Update -
We are continuing to investigate this issue.
Jan 29, 11:50 CET
Investigating -
We are currently seeing delays in sending both SMS and email messages. Troubleshooting is ongoing.
Jan 29, 11:24 CET