06 Mar State initiative prepares Wisconsin IT systems for a disaster
Wisconsin may never have a disaster on the scale of September 11 or Hurricane Katrina, but technology experts in state government are making continuity plans to ensure the Badger State responds more like New York City than New Orleans.
The state’s Enterprise Emergency Management Planning Project, conceived following September 11, was outlined during the Fusion2006 CEO-CIO Symposium presented by WTN Media last week. Those in attendance included private-sector CIOs that are well versed in the basic components of business continuity plans: defined employee roles, facility relocation, and the immediate resumption of critical business functions.
And while most attendees represented companies that have a continuity plan, questions remain about the level of executive commitment in Wisconsin, a state that experiences relatively isolated and comparatively infrequent natural disasters.
The state Department of Administration is leading the project to ensure that government continues to function in the event of a worst-case scenario – the total loss of a headquarters building as the result of a tornado or another calamity. “Until this initiative, there really wasn’t a focus on the entire state government as an enterprise,” said Gina Frank-Reece, deputy secretary of the DOA.
With its existing downtown Madison data center, DOA provides information technology infrastructure support to state agencies. As part of the project, a private company is building a new data center on Madison’s east side and will lease it to the state for $1.8 million a year.
Dave Hinrichs, deputy administrator of the state Division of Enterprise Technology, said information technology is the backbone of government service delivery in Wisconsin, and redundancy is non-existent. Connectivity between downtown, which will become a secondary data center site, and the new primary data center is being built, and the state should occupy the building in April. “And then we’d have to build it out, put infrastructure inside of the building that provides us with the hardware and software redundancy that we would need to have a very short, almost minuscule disruption in services if one of the centers goes down,” Hinrichs stated.
The DOA has identified 20 state agencies as part of the first planning group, and it’s helping the Division of State Facilities Management identify alternative sites that would provide services in the event of a disaster. Agencies will enter into mutual aid agreements to share and co-locate with other agencies; some agencies already have alternate sites that are part of their organizations.
“We’re really looking at having each agency have two designated alternate sites, a primary alternate site and a secondary alternate site,” explained Art Zoellner, director of continuity management for the DOA. “The primary alternative site would be another location here in the Greater Madison community, with the premise of the event being something that happens to that agency’s headquarters building that’s not a wider scale event in the community.”
During Fusion2006, CIOs were advised to select alternate sites that are out of the same “risk area” of the primary site, one served by a different electric and gas utility. Under the state’s plan, the secondary alternate site would be “well away” from Dane County in case of a “regional type of event, which we think is probably very unlikely,” Zoellner said.
This spring, the DOA will begin a first round of “tabletop” exercises to test the system.
Frank-Reece said the project has the full support of governor Jim Doyle, but private-sector CIOs sometimes must actively champion business continuity. Brian Jones, director of applications hosting and continuity planning for CUNA Mutual Group, is part of an organization with a long-standing corporate risk management function, including business continuity. Jones said if he was the CIO in an organization that lacked a business continuity plan, he would “push for it” with one caveat. “Is it right for IT to own that function long term? I believe it belongs in a business function, and IT is there to support them,” he said.
Another way for CIOs to get their supervisors’ attention, according to David Alvarado, vice president of storage management for CA, is to explain that the cost of planning is much lower than the cost of recovery. Alvarado also said insurance companies provide a discount on premiums to companies with a tested business continuity plan, and offer discounts for practices like using a dry extinguishing system, which reduces equipment replacement costs.
One man that has few problems getting the attention of upper management is Dennis Klinger, vice president and chief information officer for Florida Power and Light. During Fusion2006, Alvarado joked that Klinger has the “luxury” of having hurricanes blow through his state, and part of FPL’s approach was to develop a command infrastructure, including an Information Technology Command Center that measures restoration efforts and IT systems performance.
The organization continually adjusts its plan with information gleaned from post-storm reviews, and it prepares for future events with dry runs. “We’re always planning,” Klinger said. “We’re always testing.”