RCA On SSL Certificate Errors for API requests
Date of Incident: 13-Feb-2024

Incident Summary:

During the routine renewal of our SSL certificate, our team expected DigiCert to provide a new certificate signed by their existing root certificate ("DigiCert Global Root CA"). However, we missed checking the new Root certificate provided by DigiCert ("DigiCert Global G2 Root Certificate") in response to Mozilla's decision to begin distrusting older root certificates. This change was not anticipated by our team, and as a result, we overlooked the potential impact on our merchant who might not have the new root CA in their local certificate authority lists. This oversight led to SSL connection failures, affecting API calls for those clients.

Technical Root Cause Of The Incident:

The root cause of the downtime was the transition to a new root certificate without ensuring its presence in the client's local certificate authority list, leading to SSL connection issues for some clients.

What Was The Business Impact?

The incident affected merchants who might not have the new root CA in their local certificate authority lists, preventing them from establishing SSL connections.

Remediation And Follow-up Steps:

We reverted to the old certificate and are in the process of obtaining a new certificate signed with ("DigiCert Global Root CA"). We will continue to support the old Root CA  ("DigiCert Global Root CA") until our merchants are notified about the change in the Root CA.

Learnings:

• Certificate changes, particularly those involving root CA transitions, should be thoroughly planned and tested, including client-side dependencies.
• Monitoring should be enhanced to include alerts for SSL handshake exceptions in the Application Load Balancers.

Next Steps: Finalize and notify all merchants regarding the root certificate change before 2025 and their support to add the latest root certificates to their clients certificate authority list.

Conclusion:

This incident emphasized the importance of careful planning and testing when updating SSL certificates, especially regarding root certificate transitions. Moving forward, we are committed to finalizing the transition smoothly, ensuring all merchants are informed and supported in updating their certificate authority lists. This experience has prompted us to enhance our monitoring systems and processes to prevent similar disruptions, reinforcing the reliability and security of our services. We sincerely apologize for any disruption this may have caused you and appreciate your patience during our investigation.