Raising Tickets in the Self Service Portal and Email To Ticket

Incident Report for Vivantio

Postmortem

The issue relating to email processing and logging tickets via the Self Service Portal were related to a software update that was performed earlier that morning.

At 06:28 UTC on Thursday 9th September. The software release was intended to address a couple of minor bugs and through testing should not have had any wide scale impact on customer systems.

At 07:24 UTC a few customers reported issues with logging tickets in the Self Service Portal and we very quickly narrowed this down to the software release and this issue also impacted the Email To Ticket service for a a small number of customers.

As soon as we discovered this we initiated a roll back of the release at 07:30 UTC, by 07:43 UTC the rollback was showing as completed and an updated was posted on our status page, but the roll back was still in progress for some customers that prolonged the resolution of the issue.

By 08:30 UTC all customers were rolled back to the previous software version and this resolved the issues that affected those customers.

We are currently reviewing our testing and release process and we are making changes to ensure this does not happen again in the future. We are also looking at our rollback process where we notified customers too soon that the issues had been resolved.

We apologise about any issues caused and we are working extremely hard to ensure this does not happen again in the future.

Posted Sep 13, 2021 - 14:11 UTC

Resolved

The fix for the issues experienced by some customers regarding creating new tickets via the Self Service Portal and the Email To Ticket have now been resolved.

The fix has been confirmed and tested and rolled out to all customers affected.

A post mortem will be updated in due course regarding the issue and we are sorry for any inconvenience caused.

Posted Sep 09, 2021 - 08:42 UTC

Investigating

Relating to the previous issue and some customers experiencing issues with creating Tickets via the Self Service Portal.

A fix has been rolled out but it is taking a bit longer to roll out to all customers. We had previously said this issue was resolved but we have discovered that roll out of the fix is taking longer than we originally thought.

During further investigation it has also been discovered that this is impacting some customers with raising tickets via email as well. The fix that is currently being rolled out will resolve this issue as well once it has been fully completed.

We anticipate that the roll out of the fix will be completed by 09:30 BST.

We apologise for the previous notification that the issue was resolved as this was a bit premature.

As soon as this fix has been fully rolled out and confirmed we will provide an update.

Posted Sep 09, 2021 - 08:17 UTC

This incident affected: Vivantio Platform (Self Service Portal, Email To Ticket Service).