Email to ticket processing issue affecting some customers

Incident Report for Vivantio

Postmortem

Duration:
24th November 2022
Initial ticket raised 18:15 (UTC)
Further reports of issue - Incident status - raised 19:44 (UTC)

24th November 2022
Fix released 20:30 – 21:30 (UTC)
Incident status – resolved 21:37 (UTC)

Impact: Initially one customer escalated an issue related to email to ticket processing issues. Whilst investigating this issue, our engineers identified that the same issue could be impacting further customers resulting in the status page notification. The impacted customers were experiencing issues with some of their email to ticket processing. These issues were not widespread for the affected customers and was occurring for a limited number of customers.

Cause: Following an issue with the outgoing email service on 23rd November, a very small number of customers experienced an issue with the TLS handshake of Microsoft servers servicing email relay requests at smtp.office365.com.

Resolution: Our engineers picked this up very quickly and were able to easily replicate it in our QA environment and we identified that an update to one of our platform components would provide a resolution. Our engineers had the fix in our QA environment very quickly and carried out fast-track QA testing. We have a very mature CI/CD pipeline that is able to build and deploy to all of our regions and the hotfix was deployed to all of our regions within approx.. 1.5 hours of leaving QA.

Posted Dec 08, 2022 - 14:25 UTC

Resolved

Our engineers have rolled out a fix for the issue identified earlier. We will monitor the situation as per standard process however should you identify any issues in connection with the issue please let our support team know.

A post-mortem will be supplied against this incident in coming days.

We once again apologise for any inconvenience caused and thank you for your patience.

Posted Nov 24, 2022 - 21:37 UTC

Identified

Our engineers have identified a potential cause to the issue being experienced by a small number of customers currently. They are working on a fix . Once we have a further update from our team we will update this page.

We continue to apologise for the inconvenience this may be causing.

Posted Nov 24, 2022 - 20:56 UTC

Investigating

We have identified an issue affecting some email to ticket processing. This is not affecting all emails to tickets, neither is it affecting all customers.

Our engineers are aware of the issue and have started investigating.

This page will be updated once we have further information and in the meantime we apologise for any inconvenience this may be causing you.

Posted Nov 24, 2022 - 19:44 UTC

This incident affected: Vivantio Platform (Email To Ticket Service).