Hacking into SCADA

Water Plant Controls

Fortunately, for many years, the water and wastewater industry did not experience too many breaches on the digital front. But a few years back, the Stuxnet virus showed up and our digital innocence was shattered. When this happened, I was somewhat surprised because of the lack of previous attacks, but I never really heard how it all went down until last year when I sat in on a session at WATERCON 2013. When the topic came up the other day, I realized I'd never really shared exactly what I learned that day other than some rough notes I took on Cover-it-live and a general summary on the blog so figured I'd finally get around to posting that information today. 

The speaker, Michael Minkebige, a control systems engineer with Donohue and Associates started out by telling us we never really had problems in the past because hackers did not know very much about PLCs. But, he said "Stuxnet was a game changer." It was the first direct attack on PLC and HMI systems and also the first attack on governmental infrastructure from a physical angle because it was the first to destroy physical equipment. Minkebige said, "we've had other threats like denial of service but firewalls and antivirus software handled this." The other reason we were somewhat protected was because our systems and networks were old and isolated or were proprietary. He also said hackers were concentrating more on PCs – they were typically kids trying to cause trouble or break into banking systems. So basically he said we had "security through obsurity."

Then we heard about the Stuxnet attack in June of 2010. He said it was most likely deployed against Iran in 2009 by another governmental entity. Some person had picked up the virus on a USB stick and uploaded it into the Internet. There were 22,000 infections found in Iran and 6,700 in Indonesia. They suspect it took a team of 5 to 35 programmers 5 years to write the code for the virus. It is 500K bytes while most typical malware is only 10 to 15 K bytes.

The virus was spread through memory sticks and targeted Siemens PLCs and HMI software. From what Minkebige understood, the virus would "phone home" to a computer located most likely in Germany or Russia and reported what system it was on and then asked what it should do. The virus was programmed to self-destruct in June 2012. But if your antivirus found it, the virus would morph into something else. It also had two security certificates from Taiwan so it might also have appeared to be legitimate to an antivirus software program. When the virus did launch its attack it typically would change data or set points in the program. Then it would mis-report information about the operational data indicating it was operating at the correct levels or set points when it was not. A typical attack might change the speed of centrifuges by cycling them through great speed changes. There was a loss of 500 to 600 centrifuges because bearings were ripped out from this operational attack.

Unfortunately the code is now public knowledge. But there are some steps we have taken to protect our industry. Homeland Security (DHS) is on the lookout for attacks since water and wastewater plants in our country are vulnerable. And if Homeland Security recognizes an IP address from a suspect area accessing your system, the agency will notify your facility. The PLC industry has also added security to their systems. Operators are advised to keep up with patches for PLC systems. Industry organizations and societies are also publishing guidance. And DHS has released a document on how to secure your systems. We are advised to keep our systems off the Internet if possible. Otherwise, a firewall needs to be used for protection. Also, all systems should prohibit unauthorized memory devices from being used on PCs connected to your system, and you should lock out all USB connections to ensure they are not used. IT departments need to be made aware of this threat so they can monitor the systems for any suspicious activity. It's thought that future attacks might not necessarily be the Stuxnet virus, but an alteration of it.


WATERCON 2013 – Day 3

And here we are already at the third full day of the conference. Today I started out in the exhibit hall talking to folks as they walked by the booth where we had a display of the #watercon Tweets. But I didn't have much time to chat before having to run off to the Mission Impossible session. As soon as I walked in the door and was handed a sealed envelope marked "Confidential," I realized this would be no ordinary activity. Upon opening the envelope, I discovered I had been assigned to a task force to determine five action points to address signficant water-related deficiencies in a community. Fortunately I was not alone in this endeavor – there were three other water professionals on my team who were obviously very experienced. It didn't take long for us to develop our recommendations. There were three other teams in the room working on the same assignment, and when we were all done, the moderators had us share our ideas. It was interesting to see that all the teams had come up with similar approaches. Well except our team had not thought of calling the National Guard. But, hey, as the one participant pointed out – they are there and available to help so why not use them. Afterwards we were all rewarded with candy for undertaking such a challenging assignment.

WATERCON 2013 Mission Impossible session

After we were released from our task force, I managed to catch the session on Green Infrastructure. While I could not live blog it, I did manage to share a few key points through Twitter (look for the #watercon hashtag to see the stream). The overall message from the panel of speakers is that green infrastructure is made up of complex elements. Designers are still figuring things out – particularly costs. And even though we have the International BMP Database, they cautioned trying to implement someone else's solution for your project without giving careful consideration to local conditions and factors. And finally they pointed out the operation and maintenance and monitoring of BMPs can be expensive.

Later in the afternoon, I caught the following sessions and was able to live blog them. Follow the link to run the CoverItLive tool to see the main points:

Who, What, When, Where and Why of Backflow Prevention – this was also a panel discussion about cross connections and backflow protection. It was interesting to hear the approaches and ideas from each community. Several members from the audience shared their experiences too. Make sure to run the live blog tool to see the tips and advice.

The New World of SCADA Security – this session provided an interesting view of why our operations had not experienced control security issues in the past and why they are more vulnerable now. The main point was "security through obscurity." Until 2010, hackers didn't really know about PLCs or how they operated. But because several programmers got together and wrote a program to hack into a PLC, launched it on another country's operations, and had their program picked up by someone whose actions led to the eventual release of it on the Internet, now anyone can get the code. Great.

I didn't really get a chance to talk to many vendors today – the booths had to be taken down at noon. Tomorrow is the awards breakfast and a few legislative sessions. But because I have some commitments at home, I unfortunately won't be able to attend. So I guess this wraps up my summary of WATERCON, but remember, since we captured so much of it online, you can always access the information by visiting the #watercon Tweets and reviewing the live blog sessions.

I very much appreciate ISAWWA sponsoring my registration and look forward to seeing everyone back next year!


Reaction Grid: Building Community, Nurturing Business, and Throwing Tomatoes

3-D Digital Aerator for Sewage LagoonBack in 2006, I decided to join and explore the Second Life community in order to find out how it could help me as an engineer. I was also interested in finding out how it could help promote our community. As I became more involved, I realized there was even more potential to virtual worlds than I had imagined. Over the years, I eventually saw more ideas and uses emerge.

However, the main community of Second Life is not there in order to develop engineering uses for this technology. And while there are some isolated examples of people using Second Life for serious business and there are many education-based communities, there still is not a large, organized community for developing engineering-related tools in-world.

Another challenge for me has been that Linden Lab, the company behind Second Life, does not appear to be focused on the use of their technology for engineering-related work. Many have asked for the capability to import/export CAD drawings and have received little to no support. Linden Lab also seems to waver and change their terms of service a lot making it difficult for people to make commitments for its use as a design or operating platform.

I had hoped that all of this would eventually develop, but instead what seems to have happened is that a group has migrated from Second Life Reaction Grid Welcome Areaover to another “grid” or virtual world called Reaction Grid. I knew the members of this group were more focused on the use of virtual worlds for business and engineering so I visited. Now I am hopeful that I have finally found the grid I had been searching for to help me focus on the engineering and business aspects of virtual worlds.

Not only are most of the residents of this grid very intent on using virtual worlds to enhance business, but the people running the grid are interested and involved in the projects that their residents are developing and working on. They are also involved in helping all of us better understand the technology behind virtual worlds.

I think this involvement in community, which is somewhat opposite of the hands-off approach of the Second Life grid, is important for several reasons. First, if we are to leverage this technology for our work, we need to have a good understanding of how it works and its capabilities. I like that the people running the grid have invested in hosting events and classes to help us in this endeavor because I think the faster everyone learns the technology, the faster the grid will develop.

Second, what this does is encourage more residents to also get involved in helping to move development forward, not only with their own projects, but with the grid as a whole. I think the ThinkBalm Innovation Community site ThinkBalm Site on Reaction Gridon Reaction Grid is indicative of this commitment to community. This group, which is dedicated to advancement of the Immersive Internet, is based on the collaboration and sharing of ideas.

Next, some of us who are interested in using virtual worlds do not have all the skills necessary to completely develop our own projects. Working within a grid that promotes involvement makes it much easier to find others who might be willing to help.

I recently had an experience that illustrates this: In an effort to show others in my field the benefits of virtual worlds, I have been trying to set up a simple 3-D SCADA. I know it can be done, I am convinced that this is where our operating technology will soon be for our water and wastewater plants, and I know it is something to which everyone in my field could immediately relate. But trying to find a programmer in Second Life who understood what I was talking about and who was willing to do this was impossible.

3-D Fuel Facility in Reaction GridHowever, over on Reaction Grid, there were several who immediately knew what I was trying to accomplish. The owner even built a demonstration project on the grid showing how a fuel facility could monitor fuel levels and then notify operations when the tanks needed to be refilled. Now I finally feel there might be a chance I will one day help introduce virtual worlds to engineers and operators using a 3-D SCADA demonstration.

One other attraction for me has been the approach that Reaction Grid has taken to building a grid. Instead of buying land and paying tier (tax) as we do in Second Life, on Reaction Grid, you can pay to host a sim which reminds me much more of how Websites are hosted and set up. That seems like a much more viable and long-term business solution for the creation and hosting of virtual worlds. Particularly now that the hypergrid technology has been implemented. Second Life is now a walled garden that you cannot leave while these other grids allow you to move from grid to grid just like we do between Websites.

Aside from all this business, I do have to admit, there is also an element of fun to virtual worlds that increases their appeal. And attending interesting virtual events does help to further connections and community. I think Second Life has a lot to offer along these lines, and even though Reaction Grid is more focused on business, they also incorporate Fright Night on Reaction Gridinteresting builds and fun events. One of the most entertaining events on the grid has been Fright Night – an event that is set up like a drive-in movie theater where we watch great, old, creepy shows and throw tomatoes at the screen. I know that is where I will try to be every Saturday night. And whether you are interested in developing an engineering project or throwing tomatoes at a movie screen, Reaction Grid seems to be the place to be.