The Day the Data Died
I am so glad the university was not forced to use VITA/Northrop Grumman for it’s central IT support. I was very skeptical when this whole deal was announced by Mark Warner, as were some (ok, a lot) of my co-workers.
As a person that runs computers for a living, I know that restoration of just one system from backups can take quite a while (depending on what needs to be done). I also understand that restoring databases isn’t as easy as retrieving the files from a backup. Hardware issues are a fact of life in this business. But I have some serious reservations about VITA and Northrop Grumman after this.
There’s been problems in the past with missed deadlines and complaints from other state agencies about the support (or lack thereof) that they were receiving. In June it was thought that Northrop Grumman could face $5 million in penalties if it failed to resolve billing disputes and other issues with the state IT support contract. Virginia lawmakers seemed concerned the state wasn’t getting it’s money’s worth. VITA was cutting jobs. What has come of it? The state contract was extended by three years, to the tune of another $236 million. And that was after JLARC warned that Virginia should be careful and wary of privatization of more public services.
This past Wednesday (August 25), VITA found itself in the news once again. And it hasn’t been pretty. Apparently, their SAN failed. (SAN == storage area network, which translates to a central place that stores lots of files, in this case for a lot of state agencies.) The redundancy that was said to be in place also failed. The list of agencies was withheld for “security reasons and because the degree of severity varied at the impacted agencies”. The interruption was “of insufficient magnitude to activate a backup system in Southwest Virginia”. Really? So what would constitute an interruption of sufficient magnitude? An atom bomb? A category 5 hurricane? As of today (Monday, August 30) the DMV is still not able to process driver’s licenses or identification cards.
“Progress continues, but work is not yet complete for the three or four agencies that have some of the largest and most complex databases. These databases make the restoration process extremely time consuming. The unfortunate result is the agencies will not be able to process some customer transactions until additional testing and validation are complete. According to the manufacturer of the storage system, the events that led to the outage appear to be unprecedented. The manufacturer reports that the system and its underlying technology have an exemplary history of reliability, industry-leading data availability of more than 99.999% and no similar failure in one billion hours of run time.”
The comments above, which are direct quotes, make me wonder what really happened and just how much data was lost. I’m not saying that it wasn’t an “unprecedented” failure, I can’t determine that with the little bits of information that has trickled out over the past 5 days. I don’t doubt that the IT people in the trenches have been working their asses off to get things restored. What I do believe is that none of us will ever find out the truth, until that particular data is needed or someone “off the record” tells what actually happened.
The real losers so far? All those people that couldn’t get their driver’s license or identification cards renewed while the system was down will now have to prove who they are and where they live. In other words, they must renew in person, and have to bring their birth certificate, passport or other document confirming citizenship or legal residency.
This is my own personal opinion . Mine, mine and mine alone.