We've been monitoring the environment since the patch was applied, and are not seeing any login failures. We believe Teleport Cloud is functioning normally at this time.
We will begin an RCA investigation and report on our findings as soon as the RCA is available.
Posted Aug 16, 2021 - 22:31 UTC
Update
All tenants have been updated to 6.2.10, and we haven't observed any login failures since the rollback was applied. Impact should be ended.
We're currently monitoring the production environment for issues with the Teleport 6.2.10 patch.
Posted Aug 16, 2021 - 20:47 UTC
Update
We're currently updating all production tenants to Teleport 6.2.10 that is expected to fix both the login and audit UI bugs.
Posted Aug 16, 2021 - 20:12 UTC
Monitoring
We have rolled back the patch that was applied over the weekend, Teleport Cloud users should no longer see `timed out waiting for events` when logging in. However, some tenants may still see error's when using the Audit Log.
We are currently working on testing and deploying the complete set of fixes which should resolve both the problems with logging in and the issues with the Audit Log.
Posted Aug 16, 2021 - 18:56 UTC
Identified
We are working on a release that roles back the changes that affect logins, and includes a hotfix for an issue with the audit UI. We'll begin rolling out the fix to the Teleport Cloud instances as soon as the Teleport patch is released.
Posted Aug 16, 2021 - 17:50 UTC
Update
We are still working on the hotfix as the problem is currently compounded by two issues and the original patch only addresses one of the problems. Current information available suggests the login failures may be intermittent and occur in bursts, such as working for an hour, failing for an hour. We will be testing the complete hotfix shortly.
Posted Aug 16, 2021 - 16:34 UTC
Update
We believe we have isolated the root cause and are working on a hotfix. There was an upgrade deployed over the weekend but at this time we don't believe rolling back will resolve this particular problem.
Posted Aug 16, 2021 - 15:34 UTC
Investigating
This morning there have been multiple customer reports of the inability to login with the error `timed out waiting for events`. The team is currently investigating.