We noticed intermittent errors between 1:31 and 1:57 UTC. Less than 1% of the total requests failed, which occurred as one of the RDS replicas used by the Events services went out of free storage, causing database connection failures and consuming most threads in some of the Application servers.
During the incident, Our customers would have noticed a delay in webhook processing, as we had to pause the background/async activities until the fix. The failed jobs got rescheduled successfully later.
We sincerely regret any inconvenience caused to you.
Posted Apr 18, 2021 - 05:43 UTC
Monitoring
We have restored the affected services, and the systems are getting back to normalcy and we are closely monitoring the status. We are sorry for the inconvenience caused.
Posted Apr 18, 2021 - 03:13 UTC
Investigating
We are witnessing a high error rate in the last few minutes. Our team is diagnosing the problem. We will post an update shortly. Sorry for the inconvenience caused.
Posted Apr 18, 2021 - 02:00 UTC
This incident affected: Chargebee UI (Admin Console (US)), API Endpoints (API (US)), Checkout (Checkout (US)), and Self-Serve Portal (Self-Serve Portal (US)).