8.5. Mock Disaster Programs

The best way to ensure that your disaster recovery procedures are adequate is to put them to the test. Completely document several types of "disasters" and then play them out according to your disaster recovery plan. Some of these events, especially a particularly catastrophic one, take a bit more time because they may involve going to a remote location, installing servers, and performing restores of data to get things going. Not only are test procedures for recovering servers and data necessary, but you also need to take into consideration communication lines and so on. To simulate simpler problems such as hard-drive failures, you can simply have a co-worker remove a SCSI cable from your drive array. Table 8-1 shows what you could expect from a number of disaster situations.

8.5.1. Understanding fault tolerance

The idea of fault tolerance in a computer system revolves around the concept that the computer (or server, in this case) should have the capability to deal with a hardware or software failure. Probably the easiest failure to deal with is a power loss. To counter this type of failure, you can simply use an uninterrupted power supply (UPS), but would using a UPS actually constitute fault tolerance? A UPS doesn't have the capability to run forever, so you are actually merely postponing the inevitable. A better solution is to have two or more power supplies in the server that are both connected to uninterrupted power supplies. During a power outage, one UPS can supply power to the server, while the other one is charged offsite. Dual power supplies, however, brings up a topic all its own.