Editor’s note: This commentary is by Tom Evslin, the former chair and co-founder of NG Advantage, who is also an entrepreneur, author and former Douglas administration official. His blog is fractalsofchange.com.
First, kudos to the University of Vermont Medical Center for not giving any serious consideration to paying ransom, as reported in VTDigger. Even if they had trusted the hackers to unlock the files and remove all malware, each ransom paid guarantees more attacks on someone else. The hackers are in it for the money.
Second, more kudos to the staff of the hospital system who soldiered on without access to key information as round two of the virus pandemic reared its ugly head. They worked very hard to protect their patients from both dangers.
Third, though, based on public information, the hospital should have planned better for recovery from an attack like this one. They had to wipe 5,000 computers clean and put them back in service before they could use their applications again. Even a month later and with the help of the National Guard and a private security firm, the hospital had not restored full functionality and estimated the cost for each day it was down at a million and a half dollars, not counting the toll on the staff and the dangers to patients.
Planning for a disaster means having a plan which works even if the original computers have been hacked, burned, or flooded out of existence. Apparently UVMMC did not have such a plan.
At this point it would be reasonable and prudent for readers to ask whether I’m qualified to give this advice. I blog about a lot of stuff like education, politics, and economics which I’m not expert in. You don’t want to rely on amateur advice for service security.
At Microsoft in the early 1990s I was responsible for the development of server-based products including Outlook and Exchange. Later, I led the development and rollout of AT&T’s first ISP, AT&T WorldNet Service. ITXC, which my wife Mary and I founded, had a network which spanned 200 countries and provided a VoIP service despised by most of the world’s telcos and quite a few governments. It had to be hacker resistant. NG Advantage, which we also founded, has an extensive internet of things (iot) network. I’m a nerd so I was deeply involved in the technology of all these products and services.
Hospital leadership says attacks like this are inevitable; they’re right. They cite an arms race between hackers and defenders in which the good guys sometimes lose. True also. But, if you know there is a significant chance that you are going to lose access to all your servers and laptops, then you must make sure that you can restore service without those laptops and servers. The plan must be made and rehearsed in advance of the disaster. Even the “unsinkable” Titanic had lifeboats.
According to the hospital, 1,300 of the infected computers were servers – more on them in a minute – leaving 3,700 infected laptop and desktop machines. Even assuming these cost an average of $3,000 each (a lot) and assuming that all of them had to be replaced for service to resume, buying all new laptop and desktop machines would have cost only about $10 million – less than seven days of outage. Buying new computers quickly – starting with cheap ones to get back up and running – as well as a rehearsed protocol for loading all needed software onto them from somewhere other than the infected servers must be part of a disaster recovery plan. Replacing the desktop and laptop machines is actually the easy part of the recovery.
The hard part is doing without the servers which have been infected. Two parts to this:
1. Getting access to the data. Presumably UVMMC transmits a copy of its data to a location which is both physically offsite and is not part of the hospital network. I would be very surprised if they weren’t doing this. Even if the hackers locked up the onsite data, they shouldn’t have had any access to offsite data.
2. Putting the data back on servers which are not infected. As UVMMC saw, you cannot assume that your old servers will be available. Unlike the desktops and laptops, it’s not practical to buy all new servers on a moment’s notice. However, the advent of cloud computing means that you can rent the capacity of thousands of servers from providers like Amazon, Google, Microsoft or IBM with just minutes of notice and without a standby fee. You pay for and use these only until your old servers are back. Rent stops as soon as you can turn them off.
However, turning up a thousand servers in a cloud, loading them with your applications, restoring backup data to them, and putting them in use in place of your own compromised servers only works if the process has been carefully planned and practiced. Even for installations larger than UVMMC, recovery should take hours, not days or weeks – if it’s been practiced. Fatalities were high in the Titanic disaster because the crew and passengers had not had proper lifeboat drill.
I’m not writing this to be critical of UVMMC; I owe the hospital my life for their medical skill. I’m writing in hope of encouraging those who are responsible for critical IT systems in an age when attacks are inevitable to make sure that, even if there is no fool proof way to prevent all attacks, there is always a quick recovery path which does not require regaining use of the compromised computers.
See also:
