The part of our IT system responsible for deposit and withdrawal transactions of fiat and crypto currencies from and to the Main user account was upgraded to provide deposit and withdrawal operations with a number of banks and e-wallets simultaneously.
During the software upgrade, one of our servers at our primary datacenter had failed, and we switched to our fail-over server on our secondary datacenter. Unfortunately, the fail-over server suffered technical problems and disconnected from the network. Upon investigating the situation, we discovered a split-brain condition affecting the main and secondary datacenters.
To prevent any errors propagating across all instances, we decided to halt all withdrawal and deposit operations (cryptocurrencies and fiat), to return the system to a consistent state, for the purpose of which we used our backup system.
Restoring the data from the backup system and returning the system to a synchronized state took more time than planned by the upgrade.
HitBTC presents some details of its IT infrastructure to demonstrate its fail-safe and redundant structure.
- HitBTC uses Velia as its primary datacenter and OneProvider as its secondary datacenter for the purpose of replication. Each of the datacenters contains two parallel-operating instances of HitBTC’s program components.
- Moreover, HitBTC also makes use of a third facility for backup and restore in case of severe failures at any of its main datacenters.
We would like to assure you that HitBTC’s technicians always know what sequence of technical steps needed to be undertaken in hard situations.