Connectivity Disruption
Incident Report for Teleport Cloud

On May 13th at 01:24 UTC, during verification of a Teleport Cloud platform release, regional proxy services for a subset of tenants lost connectivity with the Teleport Auth service. Connectivity was restored by restarting Auth services for those tenants.

On May 15th at 15:00 UTC, a review of the incident found additional tenants experiencing similar symptoms. The issue was traced to an internal cloud component responsible for caching Teleport Auth service IP addresses to facilitate multi-region connectivity. Restarting Auth services for the impacted tenants refreshed the IP cache allowing regional proxy services to connect.

Further diagnosis produced a set of performance improvements for the IP address cache component. These action items are in progress and scheduled for release in the coming weeks.

Due to the cloud platform running Teleport in high availability mode, connectivity for the majority of tenants and users remained stable because at least one Proxy service was healthy in an impacted region.

Posted May 25, 2023 - 18:04 UTC

This incident has been resolved.
Posted May 16, 2023 - 02:15 UTC
We've taken action to stabilize tenants that were impacted and are continuing to monitor.
Posted May 15, 2023 - 21:44 UTC
We are continuing to work on a fix for this issue.
Posted May 15, 2023 - 20:14 UTC
We have identified remediation and are working on rolling out the fix for impacted tenants.
Posted May 15, 2023 - 18:54 UTC
We are currently experiencing an elevation in connectivity errors on some regional proxies. We are working towards a resolution and determining the full impact.
Posted May 15, 2023 - 16:26 UTC
This incident affected: Cloud Service.