Incident Summary:
On July 30, 2024, merchants in the AU region encountered difficulties accessing the Entitlements section of their live sites. The affected API endpoints returned a "Bad Input" error with the message: "Invalid input parameter values." This issue disrupted the retrieval of features and entitlements.
Root Cause:
The root cause of the incident was traced back to a recent system update that altered the encoding and decoding processes for input parameters, aiming to accommodate special characters. Unfortunately, the update introduced an unforeseen issue where the system failed to correctly decode these encoded parameters. As a result, the API returned "Invalid input parameter values" errors when processing numeric or Boolean input. The faulty decoding mechanism was unable to handle these types of encoded values, causing valid inputs to be rejected.
Services Impacted:
This issue affected API calls related to fetching features and entitlements, impacting both API and UI users. The problem was confined to the live environment in the AU region, while the other regions (US, EU and Test sites) remained unaffected.
Steps Taken & Future Actions:
We have fixed the decoding issue permanently, ensuring all encoded input parameters are correctly decoded during API processing. Additionally, we have enhanced validation processes to guarantee that the encoding-decoding mechanism functions properly across all input types, including numeric, Boolean, and special characters.