IT pros in south Florida were in full-on disaster plan mode on Wednesday evening with Hurricane Matthew forecasted to make landfall on the east coast of the peninsula Friday morning. This is a ten year hurricane, and it’s crunch time.
Evacuations have already begun for many residents, but IT departments are working overtime battening down the server rooms.
I reached out to a datacenter technician in Orange County, Florida for a status on the situation; he checked back with me around 8 p.m. ET Wednesday and talked about his operational protocol. /u/Justish manages IT for a multi-site small business with 50 users.
Here’s what he told me about his disaster plan.
Our servers are in a separate, secure facility from our users. We have a backup appliance taking images of our servers nightly and we send these backups to a cloud storage network outside of Florida.
If our physical server site is lost we would rebuild them with the cloud images using new hardware. We hedge our bets against everything getting destroyed, but if worse comes to worse cloud storage will still have our data.
All of our user computers are unplugged from their UPSs, everything is placed on desks, and wrapped in garbage bags. We just cover any exterior windows with tarps on the inside. Hopefully if the glass breaks it’ll keep rain out to some degree.
Large multi-function printers are unplugged and wrapped in trashbags as well. Although any amount of flooding will probably get to them.
Our remote server site will stay live unless there’s a power outage at which point the server rack UPS will gracefully power all servers down until we come in and check on them Saturday.
We anticipate losing power and maybe a facility getting damaged but we hope it’s not going to be too bad.
UPDATE: A system administrator of a 100-building public school district shared his disaster protocol on Thursday morning.
A condensed reading of how a school district IT department is handling storm prep:
- 72 hours – Directors and managers under the direction of the CIO staff must have a drafted plan in place 72 hours before the storm, essentially checklists of duties assigned to specific personnel, with an emergency phone chain to the CIO for urgent items that crop up.
- 48 hours – Begin execution of the checklists. Coordinate deadlines for data backup procedures at the Data Center and schools, and evacuation of tape backup media offsite. Open all disaster plan files as Read-Only. Designate time for mainframe shut-down, communicate this to end users. Place critical paper records on high shelves.
- 36-24 hours – Alert CIO and directors upon completion of checklists; IT staff are then to exit the building.
UPDATE 1:28 p.m. ET – Thursday, October 6: — We’ve been locked down since early yesterday. Shutters are down on the interior courtyards of the office building, and I’m down with school police in one of their offices. I’ve got a cot, and our school food services has made sandwiches and salads for everyone in our cafe. I’m thankful to have learned that we have a shower in the building. Tonight is going to be the long night, our datacenter is still up and running, we have redundant UPS systems and leadership made the decision to keep it all up, so I have to go up every few hours to make sure we don’t have leaks.
Here’s what several other systems administrators have underway:
- A one-man IT department in Dade County laments that his server rooms are prone to leakage and knows it may be game-over. He is making three tape backup copies and storing them in hard cases—one set evacuates with him, the others go with the company COO and the owner.
- A Miami area IT pro that supports a datacenter in a fourth floor suite, who performs regular backups to offsite storage, and hosts the company website out of state, is looking forward to a three day weekend. Nothing to worry about.
- Administrators in other parts of U.S. with co-located storage servers in South Florida are watching storm coverage closely; one storage technician in Atlanta in the middle of making approximately 250 node backups to a local NAS, and wishing his Florida colleagues good luck.
- One brave IT pro plans to stay on site until Saturday, sea-captain style. “Gotta make sure the data center doesn’t have any leaks,” he says.
Check out the Hurricane Matthew thread at /r/sysadmin and give the technicians a shout of support.
“Advice: If you’ve got to ride out a hurricane, do it in an internet data center…”
When Hurricane Katrina hit Louisiana in 2005, an IT pro holed up in his New Orleans data center and rode out the storm and the chaos that ensued for several days, rolling a webcam and keeping live updates on a LiveJournal page (warning: graphic language) under the username Interdictor.
We have generators and tons of food and water. It is 5 of us total. I am not sure how the internet connection will be affected. I have a camera and my gun. (Interdictor, August 28, 2005)
The mission is to keep power to the servers and water off the electronics. Go!
Uninterruptible Power Supply (UPS) is the first line of defense against power loss, but are not designed to support long term power outages. Elevate the UPS in case of flooding; they are highly sensitive to water and can spark.
Standby generators that sit outside of a building are ideal provide longer term backup power in the event that the grid goes down. They run on diesel, propane, or natural gas to feed power to your infrastructure; generators with steel enclosures offer added protection from the conditions. One IT pro in the New York area during Hurricane Sandy in 2012 said the storm blew the generators clear off the concrete mounting blocks.
Wall mount server enclosures protect infrastructure from standing water and neatly tuck away equipment in tight spaces.
Implement 3-2-1 Backup continuously
Designate all related duties to specific personnel and write it down. Rehearse and test on a regular basis. Time is of the essence, therefore everyone should know their role in a datacenter disaster plan for timely execution.