At approximately 6:50 pm CT on Monday, March 11 2019, our vendor Rackspace identified a hardware component failure on one of the redundant physical servers in our application stack. This failure allowed the server to continue to be responsive to standard monitoring queries, but degraded its performance enough that end users who were routed to that server by the load balancer experienced intermittent non-rendering of forms. All other physical application servers remained unaffected.

By 7:50 pm CT, and after working through all possible avenues to correct the degraded response time with our vendor, the decision was made to remove the server from the load balancer’s rotation. The removal of the server was completed at 8:24 pm CT and the issue was resolved.