Intermittent connection issues with outgoing email service

Incident Report for Vivantio

Postmortem

Duration:
23rd November 2022
Initial ticket raised 18:46 (UTC)
Further reports of issue - Incident status - raised 21:20 (UTC)

24th November 2022
Fix released 15:15 – 16:00 (UTC)
Incident status – resolved 16:56 (UTC)

Impact: A small number of customers experienced intermittent connection issues with their outgoing email service.

Cause: Following unexpected changes being made by Microsoft, an outgoing email component we use was impacted. This resulted in intermittent issues for some of our customers who send email via smtp.office365.com.

Resolution: As soon as our Engineers were aware of the issue and had identified the specific cause, a fix was developed which went through further QA testing and was rolled out to all Vivantio Platforms.

Posted Dec 08, 2022 - 14:10 UTC

Resolved

The Vivantio engineering team have identified that there has been a Security Protocol change on Microsoft SMTP servers and whilst Vivantio had made no changes, this Microsoft change has impacted outgoing emails that are using Microsoft's smtp.office365.com email service.
Our team have found a fix to ensure our customers are not further impacted and this has been rolled out in the last few hours.

We would like to advise you check your system health area to ensure all emails have gone out as expected. If you have any queries, please let our support team know.

Thank you for your patience in this matter.

Posted Nov 24, 2022 - 16:56 UTC

Update

Our team of engineers are continuing with their investigations into this issue. It is a priority for Vivantio to determine the root cause of the issue and as soon as we have a further update, this status page will be updated.

If you have any queries relating to the outgoing email service issue or any other query please reach out to the Vivantio Support team. In the meantime, we would like to thank you for your continued patience.

Posted Nov 24, 2022 - 10:52 UTC

Update

Vivantio engineers will be continuing to investigate this issue into tomorrow. This page will be updated further tomorrow once we have more information.

Please be assured we are investigating this as a priority to understand if the problem is with Microsoft or with Vivantio. We apologise for any inconvenience this may be causing those affected and thank you for your patience.

Posted Nov 23, 2022 - 21:49 UTC

Investigating

A small number of clients have reported that they are experiencing issues with outgoing emails. Clients that are using Microsoft's smtp.office365.com outgoing email service are experiencing intermittent connection issues. This is causing some, but not all, emails to fail to send.

Please check Admin > Setup > System Health for details of emails that have failed to send. We are continuing to investigate this issue however the possibility exists that the problem may be at Microsoft's end and outside of our control.

Posted Nov 23, 2022 - 21:20 UTC

This incident affected: Vivantio Platform (Main Vivantio Application).