GitLab system status: Operational 🎉 @GitLab Twitter handle status: not so much ☹️ If you are reaching out to @GitLab for assistance, you may get a response from this handle (@gitlabstatus) or from one of team members' Twitter accounts.

We are currently investigating health check availability issues with our API fleet on GitLab.com. Follow the working doc at docs.google.com/document/d/1… Issue at gitlab.com/gitlab-com/produc…

We've restored service to the GitLab.com container registry, registry.gitlab.com. Post-mortem to follow at gitlab.com/gitlab-com/infras…. Apologies for any inconvenience caused.

We have resolved the performance degradation GitLab.com was experiencing in Postgres queries. We will continue the post-mortem will be conducted in gitlab.com/gitlab-com/infras…. Apologies for any inconvenience caused. monitor.gitlab.net/dashboard…

GitLab.com is experiencing high latencies due to Postgres database query issues. We are currently investigating why some queries are no longer being optimized correctly. monitor.gitlab.net/dashboard…

GitLab.com is operating normally once again. We are continuing to investigate the root causes of today's incident, and will conduct a post-mortem in gitlab.com/gitlab-com/infras…. Thanks for your understanding, and once again, apologies for any inconvenience caused.

Service is returning to normal on GitLab.com after our secondary instances lost replication while carrying out planned maintenance. Retrospective to follow at gitlab.com/gitlab-com/infras…. Apologies for any inconvenience.

GitLab.com continues to experience high latencies and queue times after a corrupt replication segment caused our secondary Postgres instances to stop replicating off the primary. We are working to resolve the problem. Working doc here: docs.google.com/document/d/1…

Initial investigation indicates that during a deployment, a backward incompatibility resulted in cache poisoning. This caused a sharp increase in git errors between 09h35 and 10h05 UTC: Postmortem: gitlab.com/gitlab-com/infras…

Live stream of us working to restore GitLab.com: youtube.com/watch?v=nc0hPGer…

Memory increase seems to have had quite the positive impact on API (1) and SSH key looks (2) timings:

Still early to tell for certain, but the impact appears to be quite promising.

We removed failed machines in shared runners pool. We will be monitoring the performance for a next few days.

Rebooting it from AWS