13 Aug Alliant Energy's flood response offers IT business continuity lessons
Madison, Wis. – David Cagigal now has lived every CIO’s worst business continuity nightmare. But flooding that hit Alliant Energy Corp.’s Cedar Rapids, Iowa facility in June, when the Cedar River crested at 32 feet, gave Cagigal’s IT department a chance to demonstrate its value.
While the electric and gas utility still has some recovering to do, IT passed the test thanks to a well-defined business continuity plan, a workhorse backup generator, and a partially completed data migration.
Alliant avoided the 20-cent per share financial hit it had announced early in the crisis, and information technology is viewed in a new light. Cagigal said IT rose to a new level of importance in the organization, and CEO Bill Harvey spread praise around during a recent investor conference call.
“In spite of the fact that five of our facilities in the Cedar Rapids area were evacuated as a result of the flooding, including our corporate offices downtown, our planning and execution ensured that our operations continued,” Harvey said.
Still, Cagigal can point to a number of IT lessons learned, including the importance of data replication. In this look at Alliant’s approach to business continuity, WTN found several important CIO take-aways.
The storm swept across Iowa on Wednesday evening, June 11. While flooding caused 75 percent of the counties in Alliant’s three-state territory to be declared federal disaster areas, the most severe damage was in Cedar Rapids, Alliant’s largest load center.
Cagigal was in Madison receiving storm updates via cell phone when he dispatched an IT team leader to Cedar Rapids for a first-hand account. When she arrived at midnight, water was flowing over the riverbank and toward Alliant’s 21-story Cedar Rapids tower.
By Thursday, the first floor was under water, as were 1,700 city blocks. Service to 35,000 customers was disrupted. Alliant’s 185-megawatt and 55-megawatt generating stations were knocked out; the utility still is working to bring them back.
A myriad of steps were taken just to keep Alliant running: the relocation of infrastructure services, a virtual server environment for metering applications, special phone and data systems for customer billing, and temporary work locations.
The main data center and network operations center, both located on the tower’s fifth floor, were inaccessible. Alliant had been using the Iowa data center as a redundant backup for Madison, and was setting up the reverse when the storm hit. Information technology systems were restored within five days; during those five days, the data replication project helped avoid service disruptions.
It also enabled Alliant to restore e-mail and other communications in a matter of hours, not days. As equipment was being shipped from Madison to the tower and temp sites in Cedar Rapids, Alliant conveyed from Cedar Rapids to Madison network shared drives, e-mail servers, and voice-mail servers. They were needed in the Madison data center to make possible communications in Iowa. By Monday, Alliant established power in Cedar Rapids with a diesel-powered generator, and the drives and servers were transported back to Iowa.
“Otherwise, all the other data was replicated and sent over an OC-48 [broadband] line from Cedar Rapids to Madison,” Cagigal explained.
Alliant expects to complete data replication in October. Without Madison as a live backup, Iowa customers would have faced prolonged outages and, now, all of Alliant’s single points of failure are being reviewed.
Some luck was involved. Workers found an entire bank of modems and work stations on the 18th floor, and the modems were used for meter reading. The modems since have been consolidated into the data center, expanding redundancy.
Members of Alliant’s crisis management team set up eight temporary work locations, including Kirkwood Training and Outreach Services (KTOS), a college that served as the emergency operations center. The facilities were provided with the good graces of other organizations, not as part of a contractual relationship.
Since KTOS had no equipment or network connectivity, Alliant dispatched laptops and other equipment from Madison, and had a local vendor drop in a cable modem to establish connectivity and a virtual private network. The improvised network had more than 450 temporary network connections and over 200 VPN connections for employees’ home use.
Also deployed were seven servers, 200 VoIP phones, 146 replacement laptops, 60 new and 27 temporary cell phones, and 20 new BlackBerrys.
Alliant benefited from its practice of “life-cycling” PCs, which provided an inventory of equipment. Vendors pitched in by providing people to handle the dispatching of equipment, Cagigal said.
The company’s business continuity plan, which employs calling trees in an emergency, has well-defined roles in people’s names, and it’s documented and rehearsed. “What we didn’t do in those plans is identify the temp locations,” Cagigal acknowledged. “I think we deliberately choose not to do that because along with that comes contracts and hot-site obligations and invoices to be paid.”
The risk associated with not having such contractual assurance is something Alliant will review. With business continuity, Cagigal said it’s important to assess the risk of a reactive plan versus a contracted plan. Cedar Rapids, he noted, is a small town with intimate relationships.
“In a larger city, with many companies suffering the same outage, we would be queued up and taking on a significant amount of risk of not being able to run our business,” he said.
Cagigal said the willingness to suspend policies and controls will be factored into his evaluation of lessons. Without abandoning the usual purchasing procedures, Alliant would not have been able to immediately purchase office supplies or T1 lines; the process would have taken weeks. This is another area where vendors had a chance to shine, as both AT&T and McLeod (now owned by Paetec) circumvented some of their controls.
“We are evaluating our lessons learned in their entirety,” Cagigal said, “not just IT but business processes and availability of people.”