During the datacenter migration a handful of transactions were recorded in the old datacenter and stranded there. Maintenance to pull those transactions into the new datacenter inadvertently disrupted some transaction processing from April 13th 5PM Central until April 14th 1AM Central.
The problem transacting was discovered during the 1AM maintenance window intended to clean up errors in the imported records.
The team spent this last weekend assessing what donors had missed a giving opportunity, which donors had returned to try again after the problem was corrected and assessing the data integrity in general.
Work continues to insure no further instabilities result from any further corrections, cleanup and migration errata.
During the cutover from Rackspace to AWS, 31 transactions were stranded on Rackspace.
Following the cutover, AWS reassigned their IDs to other, new transactions that were processed on our AWS servers. To migrate these 31 transactions new IDs were assigned that would not interfere with other transactions already in AWS. This reassignment of IDs, though tested, created errors in the platform and additional, new records were assigned subsequent IDs that also generated errors.
During maintenance last night the migrated records and all records created after that were reissued new IDs and the database instructed to reset the ID assignment.
At this time an error exists somewhere in the reassigned IDs preventing them from being displayed in reports or in the control panel. We are identifying the error and developing a plan to correct this, but in the interest of stability over the weekend, we will not be performing any further database work until next week.