What Happened

On Friday, March 22, 2019 at 2 AM CT, recurring gifts failed to process for our Enterprise customers. Roughly 3,200 gifts were unable to transact according to their normal schedule for that day. Our logs indicated that the communication from our servers in our data centers hosted in Rackspace failed to communicate with a number of our customers’ payment processing gateways. Unfortunately, our monitoring systems did not alert us to this issue since the first recurring gift was successful but all failed thereafter, and our standard reporting mechanism did not see the large discrepancy and report this to us.

At roughly 10 AM CT that morning, an Enterprise customer alerted our support team of suspected issues with recurring gifts not running as scheduled. This was immediately confirmed as an issue and at 10:06 AM CT our team started the recovery process and communicated to customers that the team was working on investigating issues with recurring gifts.

We are still investigating the root cause of the issue as to why the servers failed to communicate successfully with the payment processing gateways.

How We Resolved the Issues

At approximately 10:49 AM CT in an attempt to re-run the full schedule of recurring gift transactions that had not processed, our team members inadvertently ran the recovery process twice in quick succession. Our built-in duplication prevention mechanism was able to prevent approximately 400 gifts from processing twice; however, 2,800 of the 3,200 were duplicated.

At 10:51 AM CT our team updated the status page (http://enterprise.givegabstatus.io/) to reflect this inadvertent duplication of gift processing and started to plan out the voiding and refunding for the duplicate charges.

At 11:00 AM CT donors started to receive duplicate email receipts and shortly thereafter we started to receive reports to our support team. The team continued to build a report of all the duplicate charged transactions and impact per customer.

At 12:30 PM CT, we sent an email to all Enterprise customers making them aware of the duplicates and that we were in the process of refunding all charges. Additionally, we started to work directly with our key partners to communicate with them the issues as well as discuss the preferred communication strategy with their donors. Our support team then began to manually process refunds for all impacted donors.

The team continued processing refunds into the evening of March 22nd. The intent was to process as voids within 24 hours if possible so that charges did not appear on donor credit card statements and so that customers did not incur refund charges.

We successfully processed all refunds within 24 hours. In some cases, voids were not possible, so some customers may see additional refund processing fees, which GiveGab will cover. The support team will follow-up on this with certain customers.

The team continued to monitor all recurring gift schedules and processing each night from March 23rd to March 25th and all successfully processed normally.

At 9:43 AM CT on Monday, March 25, 2019, we sent an email to all Enterprise customers making them aware that all duplicate charges were refunded.

What We Are Doing to Prevent this Going Forward

The team is looking at a number of things to prevent this from happening again, including:

1. Enhancing our internal processes to ensure multiple team members sign-off and check before we take corrective action on re-running recurring gift schedules

2. Adding better monitoring to alert us early on for discrepancies surrounding failed transactions, not just if the batch fails in total

3. Add confirmation prompts to the re-processing function to ensure support staff have double-checked that they still need to run it

4. Looking into a better locking and duplication prevention system for more immediate detection of quick successive runs of the re-processing function

Summary

We sincerely apologize for the inconvenience that this incident caused to our customers and their donors. We are dedicated to improving our processes and software so that we can avoid this from happening again. Please do not hesitate to reach out to enterprise-support@givegab.com if you have any further questions or concerns.