RCA On NMI Payments Service Disruption
Date of Incident: 21-Feb-2024
Incident Summary: Certain merchants experienced transaction failures in the NMI payment gateway with the error message "Custom descriptors are not allowed for this processor" following a release deployed on February 21, 2024. The issue stemmed from Chargebee initiating statement descriptors to the NMI Gateway, resulting in 4xx errors. Despite thorough testing in sandbox environments, the errors only surfaced after the release was deployed to production.
Technical Root Cause Of The Incident: The root cause of the incident was traced back to a recent update in the statement descriptors implemented by Chargebee to the NMI Gateway. This update inadvertently triggered conflicts with the processing capabilities of NMI for certain merchants, resulting in 4xx errors occurring during transaction processing.
The underlying issue stemmed from a flaw in the system's validation process; our system didn't properly check if the statement descriptors were compatible with all processors supported by NMI as per Advanced Merchant Features (AMF). This oversight meant that some merchants were affected without us realizing it. Ultimately, this service disruption caused payment failures.
Though we have monitoring in place for server-side 5xx errors, the absence of appropriate monitoring mechanisms for 4xx errors further delayed the timely identification and resolution of the issue.
What Was The Business Impact? The incident resulted in payment failures, causing disruptions in the checkout, payment collection and renewal workflows.
Remediation And Follow-up Steps: To address the issue, Chargebee took immediate action by disabling the statement descriptor feature for the NMI gateway across all merchants. This feature remained disabled for all merchants using the NMI gateway, and re-enabling it was made available only upon request. Additionally, we implemented relevant alarms to enable swift detection and notification of similar incidents in the future. These measures ensure that we are better equipped to respond promptly to any potential issues, thereby minimizing disruptions to payment processing and safeguarding merchant operations.