Summary

Teleport Cloud customers experienced connectivity issues between the hours of 17:10 UTC and 23:14 UTC on 2022-06-08. Increased memory consumption by Teleport Cloud Worldwide components triggered the outage.

Impact

An estimated 23% of Teleport Cloud customers experienced connectivity issues between 17:10 UTC and 20:20 UTC on 2022-06-08. 

All Teleport Cloud customers experienced connectivity issues from 21:00 UTC to 23:14 UTC on 2022-06-08.

Root Cause

Increased memory consumption by software that enables Teleport Cloud Worldwide triggered the initial connectivity issue. This memory consumption interrupted adjacent workloads on the same host. In order to alleviate the load, a core component of our infrastructure was upgraded which caused a service interruption for all users.

Recovery

The initial connectivity issue was resolved by scaling Teleport Cloud. The failed upgrade of the core component was resolved by rolling back to the previous version. The extended outage was due to incompatibility between versions.

Corrective Actions

1. The capacity of the Teleport Cloud fleet will be increased in all regions during the next maintenance window.
2. Monitoring service health will be modified to enhance early detection.
3. Enhancements scheduled for Teleport 10 will reduce memory consumption for Teleport Cloud Worldwide components.