Issue with logging into Vivantio

Incident Report for Vivantio

Postmortem

Between 23:00 UTC on 03 Apr 2024 and 03:29 UTC on 04 Apr 2024, Database services in the Azure US East region experienced connectivity issues. The Vivantio Platform for US customers is hosted in the Azure US East region and was impacted by these issues. The connectivity issues were caused by the deployment of front end gateways by Microsoft into that region. Microsoft mitigated the issue by stopping the active roll out deployment.

We apologise about any inconvenience caused and thank you for your patience while we worked through the outage. We will be conducting further root cause analysis on the issue and working with our data center partners to mitigate any future issues of this nature.

Posted Apr 04, 2024 - 08:20 UTC

Resolved

This incident has been resolved.

Posted Apr 04, 2024 - 07:59 UTC

Update

04/04/2024 03:20 UTC Update
All Vivantio Services are operational again. Engineers are still closely monitoring services and working with our data center providers on the root cause. Further information will be provided in due course.

Posted Apr 04, 2024 - 03:20 UTC

Monitoring

04/04/2024 02:45 UTC Update
Most services are back online for those impacted customers. Engineers are still working on the issue related to the Azure SQL Database instance but the majority of services are back online. We are continuing to monitor the situation and work with our data centre partners. Once the issues is fully resolved we will provide further information about the root cause.

Posted Apr 04, 2024 - 02:47 UTC

Update

04/04/2024 02:00 UTC Update
Engineers are still working on the issue which is related to a specific Azure SQL Database instance. Any customers who are hosted on that Azure SQL instance will be affected. The availability of the Azure SQL instance is at a status of degraded performance and has recovered from an unavailable status so customers may find they can login in intermittently.

An update will be posted as soon as possible and by 03:00 UTC if not sooner,

Posted Apr 04, 2024 - 02:04 UTC

Update

04/04/2024 01:31 UTC Update
We are working with our data center partners and we have engineers investigating the issue. The problem is affecting all customers within a specific Microsoft Azure Region and will result in the affected customers not being able to login to their Vivantio system.
As soon as we have further information we will post an update here, the next update will be due at 02:00 UTC if not sooner.

Posted Apr 04, 2024 - 01:35 UTC

Investigating

Vivantio have been made aware that some customers are reporting an issue with logging into Vivantio. The issue has been escalated to the Vivantio engineering team and they are beginning investigations.

This page will be updated at regular intervals but in the meantime we apologise for any inconvenience this may be causing our customers.

Posted Apr 04, 2024 - 00:54 UTC

This incident affected: Vivantio Platform (Main Vivantio Application, Self Service Portal, Email To Ticket Service, Reporting and BI Services, Scheduling Service, API and Integrations).