After a successful deployment of a product update at 11:13 PM ET, we confirmed that transactions were occurring successfully and began our routine post-release manual testing of the platform.
At 11:19 PM ET, our internal testing revealed degraded performance for newly deployed code that was intended to better leverage our caching vendor’s services. This degradation resulted in inconsistent form rendering and we immediately began the process to roll back to the April 14, 2020 release version.
At 11:32 PM ET, the rollback was complete, transactions continued to be taken and the form rendering returned to the expected consistency.
During this 19 minute incident Forms that had been loaded prior to the completion of the update would submit successfully. However, new requests for forms not previously loaded prior to the update would sporadically render. It is possible that donors/registrants may have been shown a form, but then not shown a form after a browser refresh, or vice-versa.
Posted May 21, 2020 - 00:03 EDT
This incident affected: Core Platform (Donation Processing Forms).