In the summer I escape Texas heat to vacation in cool elevations of northern New Mexico. It is a 12 hour drive each way. Fortunately, I have lots of podcasts I can listen to. One of them is "Stuff You Should Know". The hosts Chuck and Josh have recorded over 1,000 shows since 2008, each of which runs about an hour, and they cover a very broad range of subjects. Cruising through the broad expanse of west Texas, I was delighted to listen to their August 17 broadcast covering cyber security for SCADA (Supervisory Control and Data Acquistion) systems entitled "Are We in a Cyberwar?". This was a subject topical to my work and to this newsletter.
Chuck and Josh begin their story in Bellingham WA. On June 10, 1999, there was an explosion at the Olympic Pipeline facility that killed 3 people, injured 8, and resulted in USD 45 million in property damage. Of the 5 causes identified by the National Transportation Safety Board (NTSB), one of them was "performing database development work on the supervisory control and data acquisition system while the system was being used to operate the pipeline, which led to the system's becoming non-responsive at a critical time during pipeline operations." This incident became the subject of the hyperbolic April 9, 2008 article in Wired magazine entitled "Industrial Control Systems Killed Once and Will Again, Experts Warn".
The reason the story begins here is that this incident is recognized by many as the first case of a fatality in which a SCADA system bore some responsibility. Chuck and Josh say that this caused many cyber security experts to become aware that SCADA systems could potentially be used to create disasters.
The next chapter in the story is the 2010 Stuxnet incident. This was a successful effort, likely developed by US and Israel, to infect a SCADA system used in Iran's nuclear program. Stuxnet was able to disrupt the frequency of drives controlling uranium enrichment centrifuges by infecting controllers in the SCADA system, which damaged Iran's nuclear program.
I recommend listening to the rest of the story as told by Chuck and Josh, but I will now take this opportunity to make some corrections that need to be taken into account. I do not expect Chuck and Josh to master every subject on earth, so I was not surprised to find a few problems in their podcast
* In the podcast they refer to a SCADA system as "SYSTEMS Control and Data Acquisition System". The correct derivation of the acronym is "SUPERVISORY Control and Data Acquisition System". Perhaps a semantic correction, but noticeable for those of us in the business.
* More substantive is the claim that SCADA systems run Microsoft Windows. While this is true, it can lead to the wrong conclusions. In our industry we have SCADA and DCS (Distributed Control Systems). These systems originated with different designs but continue to converge technically. While the computers that provide graphical displays, history, and higher-level functions predominantly run Microsoft, controllers and IO processors don't. The devices close to the field that have logic and control algorithms require a realtime, deterministic operating system. Therefore, they don't run Windows. SCADA began as a combination of a PLC (Programmable Logic Controller) running its proprietary realtime operating system with a PC running a commercial operating system (such as Microsoft DOS or Windows) to provide engineering and operator functions. DCS design began as a proprietary system throughout for larger scale systems where deterministic processing was required at every level. Over time the DCS has migrated from proprietary to commercial (Microsoft) computers with the exception of the field controllers/devices. If we combine SCADA and DCS and refer to them as different types of ICS (Industrial Control Systems), we can say that an ICS is neither wholly Microsoft nor wholly proprietary. It cannot be said that ICS field devices have the same vulnerabilities of a computer running Microsoft.
* The blame on Microsoft is further refuted by the fact that the incident in Bellingham that began this story was not on a Microsoft system. The system at Olympic Pipeline ran on a VAX computer running VMS operating system from Digital Equipment Company (DEC). In 1984 I began my career in the paper industry working on an ICS using an earlier version of this DEC platform, a PDP-11 machine running RSX-11M operating system. I can attest to times that we had problems with that ICS, so Microsoft is not a common denominator to every control system malfunction.
* The incident in Bellingham was not a cyber-attack. Chuck and Josh referred to it as a system malfunction. It is not clear that the Olympic Pipeline ICS had any malfunction. It is possible the user that made online database changes causing the system to be nonresponsive, as suggested by the NTSB report. System failures can happen regardless of a cyber-attack, and some failures are human caused.
* Bellingham illustrated what can happen if an attack on an ICS occurred. Chuck and Josh say that the US is particularly unprepared because we adopted connectivity to the Internet so quickly while cyber security lagged. This suggests that all the threats come from connectivity to the Internet. Wrong! Stuxnet happened in Iran on a system with no outside connectivity. No hacker intruded over an Internet connection. Stuxnet got into that system through a USB drive. It is unclear whether a secret agent inserted it or if USB sticks deliberately dropped in a parking lot ended up being used by someone on site. It is well known that if someone sees a USB laying on the ground, they are likely to pick it up and use it.
* Chuck and Josh say that Stuxnet infected Siemens switches. The infection was in Siemens controllers, not switches.
* The greatest threat to your system is not Internet connectivity. It is serious, but not as likely as some human in your facility. Whether deliberately or accidentally using an infected USB or someone willfully walking into a room with no locks or security on the doors and taking out their grievances by using your system to cause harm is statistically a much more likely risk. If you have no policies on who has access to a room, who has login credentials, and how peripherals like USB or laptops can connect, you can lockdown your outside connectivity and still be at risk.
* If you still think air-gapping your system (completely disconnecting it from the outside) will make your immune, consider that such a system requires a user to copy files by connecting USBs or laptops. Therefore, you may make your system more vulnerable with an air gap than by having a connected system built for security.
* Chuck and Josh state that SCADA systems are the Achilles heel of our infrastructure because of their cyber vulnerability. Wrong! If you want to weaken your facility, remove the ICS. That is why I call the article by Wired magazine hyperbolic. Nearly every industrial facility in the world has an ICS because it makes us safer, cleaner, and more sustainable. It can always be argued that an ICS should be safer, but not having one would be unsafe.
* Chuck and Josh state that SCADA systems are used in the US, and therefore the US is more vulnerable. Wrong! Industrial facilities worldwide require an ICS for the same reasons as stated above. The US is no more vulnerable because facilities have an ICS.
* Chuck and Josh state that the US is lacking in preparation for cyber war compared to countries like China, Russia, North Korea, and Iran. I see no evidence of this. Iran is less connected than the US, yet Stuxnet got into their disconnected ICS.
* Chuck and Josh didn't know where the term "ping" comes from. Ping is a common way we determine if a computer is connected on a network. The origin is from active sonar used on submarines to send a "ping" through the water to see if the signal reflects off another vessel. If the sub detects the signal bouncing back, it knows there is another vessel nearby.
Despite this long list of corrections, clarifications, and opinions, I am not hating on Chuck and Josh. I will still keep listening and recommending their show.
My recommendation to you is to seriously look at your vulnerability. I may have sounded like I was defending Microsoft, but we all know that hackers target the most popular operating system in the world and there are vulnerabilities there. I have been to a lot of your mills. Many of you have doors wide open to these risks. I also do projects on critical infrastructure for the US Federal government, municipal governments, and private industry and many do not comply with fundamental standards of security or their own policies. Chuck and Josh are correct in stating that the US is very vulnerable, but it is not all due to Microsoft, Internet connectivity, and none of it should be blames on an ICS. We create our own vulnerability by failure to defend ourselves.
Some of you have recovery boilers. I hope I don't have to remind you that they can be dangerous. Even if you don't have such a dangerous process at your facility, hackers are attracted not just by causing a disaster. They want money and can hold you hostage for ransom. On a recent project I was one of 10 engineers working on a SCADA system that was only going to be in development for just 2 months before being shipped to a site. Many of the engineers needed to remotely connect from their location to do their development, so connectivity was required. The facility that staged the system decided that since the system wouldn't be there very long, they would not put in a firewall. After 6 weeks of work, we were all suddenly taken offline when an intrusion was detected. Someone from somewhere got into our system and started deleting files with a ransom message. We were able to remove connectivity quickly enough to protect the 6 weeks of work that our team had completed. We had to rebuild servers and restore some of our work without having to pay ransom. We were very close to a very costly failure to protect ourselves that could have resulted in a project failure to meet schedule and budget. The lesson is that defense is not optional.
In closing, I want to share the good and the bad.
Bad news: Your production and safety are vulnerable and can be exploited by those seeking to hold your hostage for ransom.
Good news: Podcasts are an enlightening and entertaining way to spend 12 hours in a car
