While this story is getting a bit dated, the timing for my article now is intentional. As you may have seen recently, CORE Security released a cyber vulnerability notification for a problem found by one of it’s analysts in a CITECT product, http://www.coresecurity.com/?action=item&id=2186.

This leads us to question whether or not vulnerability disclosure is the right thing to do or not for the SCADA and process control industry. Of course this question comes up time and again for us here at Wurldtech as well. Hardly a day goes by that a vendor or asset owner asks us if we are going to release the vulnerabilities we find in industrial control systems, and we find a lot. Vendors are concerned we will publicly release the data without giving them a chance to review it. Asset owners are split, some want us to release, some do not. Those that do want us to publish as widely as possible so that they can get the vendors to deal with the issues. Those that do not are concerned about protecting their own facilities.

One thing is for sure…. perhaps two things:
1- Releasing or not releasing, no matter what you do, a massive amount of people will be mad. It seems to me that NOT releasing probably has a lower anger threshold than releasing, and is also a more ethical thing to do. More on that later where I will expand this point
2- No matter what we do (or not do), SOMEONE will release vulnerabilities. For those that don’t know the security business, many “white-hat” or “gray hat” firms actually purchase vulnerabilities from crackers that wish to remain anonymous. Sometimes these crackers have altruistic motives (aka whistle blowers), and sometimes they just want to get paid. Either way, vulnerabilities have come out and will continue to come out.

Did CORE do the right thing? I am sure we can debate this for a long time. They are a respectable organization and they do some great work, and they are free to do what they think is right. I have no idea the extent to which they may have worked with the vendor ahead of time. Not my place, nor any of ours, to decide what is right and wrong.

Here is the problem: Vulnerability disclosure is the norm in the IT world, not the exception. If the good guys don’t do it first, then the bad guys will. I would much rather the good guys be the release agent, AFTER they have worked with the vendor and given them a chance to respond.

Before someone jumps to the conclusion that I am saying we should release vulnerabilities, and wondering when Wurldtech will, don’t jump the gun. There are a few factors here to consider:

1- Industrial cyber security is still an evolving discipline and lots of lessons to be learned
2- Fixing vulnerabilities is not so easy, patching just isn’t the normal mode of operation (yes, I know many companies have found a way to do it, that is not this discussion at this time).
3- Hackers/Crackers are not, for the most part, distributing vulnerabilities and security hacks for control devices, yet. That is changing, and will change, but why should we rush the process?
4- Vendors and asset owners still are coming to grips with how to deal with these issues
5- Solution providers still have a ways to go to offering solid, comprehensive cyber risk mitigation strategies.

All of this is coming, and it is an exciting time for sure. We are definitely carving new snow and defining ways to address problems that were previously not even imaginable, let along achievable.

Is releasing the right thing to do? I will not try to decide, nor judge, if these companies have made the right choice by disclosing or not. What I CAN say is that it is not the mode of operation for Wurldtech. We think it is more important at this time to increase the body of knowledge about security failure modes, build resilience profiles for broad categories of devices, and work with vendors and asset owners to define intrinsic security fixes, compensating controls, better network management, and process optimizing solutions that target the things that operators and owners really care about: keeping operations running and efficient, period.

The Achilles Delphi program is a great example of this leadership. We are already testing and actively building this knowledge, and we are working to put together actionable information in the hands of those that need it. We are not releasing vulnerabilities as part of this effort, but rather mitigation solutions and things that vendors, solution providers, and asset owners can actually run with.

Will we someday release vulnerabilities? It will depend on market demand, and the general level of capability in the market. We have been very critical of those in the past that have release vulnerabilities, largely for the reasons above. One can rest assured, however, if we every DO decide to release, it will be done to the highest professional and ethical standards, and it will only be done once vendors have had an opportunity to work out the issue, and we know a mitigating solution to the problem. It certainly will not be for “the glory of the 0-day.”

As I have said now many times… we are not in the business of finding problems just so we can get paid to fix them. Wurldtech’s mission and focus is to improve the reliability, resilience, and security of industrial devices and SCADA systems, and raising the bar for the industry to drive secure, efficient, and benefit driven solutions to our customers, period.

As some of you may have been watching on the SCADASEC email listserv recently, there has been a bit of an uprising… again. Pondering this in a semi satirical state last night (after about the 20th email sent and received on the issue), I started wondering why this happens so often. It finally hit me… we have two very distinct camps on polar opposite ends of the spectrum. Let’s call them pneumatic pumpheads and bit-twiddlers. Now, before any feathers are ruffled, keep in mind that this post is equal-opportunity - everyone is going to be given an equal ration of grief…

For the purposes of this post, its VERY important to remember that everyone has a lens through which they view the world. Even using the same terms and language, people can see different messages because of the lens that they use, it’s unavoidable. So, while we are reading this post, let’s get our prescriptions checked and try to think in as much of an unbiased way as we can.

Pneumatic pumpheads represent the “classic” versions of engineering. They are mechanical, electrical, chemical, civil, logistical, and just about any other version of engineer out there. They often represent folks that have lots of experience and knowledge about any given process, but they also often are locked in the ways that they have always thought. If you have every been to any type of plant, you recognize the person immediately. They are the ones that can crunch calculus in their heads, work a calculator like no one’s business, but they are also the ones that look down their nose at computers and constantly complain that the computer, “just doesn’t work.” Their knowledge of engineering and process is often first rate and of vital importance, but many just don’t want to know about computers. They are often very precise, very literal, and very utilitarian when it comes to solving very complex problems. But, they are much happier to solve the problem with a wrench, blowtorch, and good ‘ol fashioned sweat equity.

Now, bit-twiddlers. These are the folks that think in 1’s and 0’s, they see the world through computer logic, state machines, and that nothing is “absolute” quite often. These are the folks that bring us things like linear algebraic solvers to optimize a blending process, monitoring and control solutions that allow one person to support eight plants, and would rather sit at their desk than don fire retardant clothing, hair nets, steel toed boots, and venture into the plant. They often arrogantly view engineering as a “subset” of computer science, and don’t understand why distributed control can be a bad thing (tell an operator of a turbine or a boiler that you want to make changes from three states away, and you are likely to learn a few new words from him or her). To these folks, they can not understand uptime, availability, etc. They would rather sit at a keyboard and direct the plant. They are into problem solving, but in a more abstract manner. Problems to them require elegant solutions, and they are more likely to write a logic tree, decision support matrix, or some kind of statistical solver to deal with natural variability in a process. There are no absolutes to them, only optimization. Theirs is a discipline where elegance often wins out over practicality, and critical issues such as safety. I’m reminded of an auditor a few years back that told me (after they shut down the plant and created several major issues from a network scan) that their way was much faster and they would keep doing it the same way from now on. She just refused to believe that these were possible safety issues… her lens told her that there was no way that a network scan could ever hurt someone.

Arrogance abounds on both side of course, and these are definite polar opposites. Many people sit somewhere in the middle. Myself, I’m an IT person (software developer and systems admin) that happened to land in engineering and industry. So, I probably fall somewhere in the middle. The modern plant is developing more of these middle ground folks, it is where we have to go. Some schools are offering courses at the bachelors and masters level now in “mechatronics” which fuses engineering disciplines with IT and computer logic disciplines. This IS the plant of the future.

So, its not an “either / or” but rather a “both / and.” We will not solve the challenges of today by thinking like the engineers of yesterday, and we can not arrogantly assume that the cutting edge of computer systems and technologies can solve all problems. Students coming out of school today and folks that are coming up to replace the aging workforce have more computer skills than ever before. Economics and the need to compete in the global economy is driving the need for more efficiencies, and is casting “one-trick ponies” off to the side. If you only understand one discipline, don’t be surprised when you job moves overseas and you are left in the easy chair complaining about it being unfair. The world is a different place and we all must be ready at any moment to reinvent, diversify, and lead.

In the not too distant past, cutting edge western medicine explained illnesses in terms of humours.  If you had a cold, you had too much phlegm, so you would balance your humours by increasing your yellow bile, which was antagonistic to phlegm.  Apparently this involved sitting in bed and drinking lots of wine.

Now, as comfortable as this remedy sounds, it has a drawback: it doesn’t work.  The idea of humours has some correlation with reality, since it was based on observation, but it is oversimplified.  Now we know that the outward symptoms of colds are our body’s attempt to remove an invading virus.  Our knowledge is still far from perfect, but we know why there’s all the mucus and sneezing.

The important difference is that we now view the human body as an extremely complex system, with failure modes and compensation strategies all built in.

Our current view of network security and reliability is very much like the Hippocratic view of medicine.  If you have hackers, you have too many vulnerabilities, so to counter them you get patches and firewall rules.  Apparently this involves buying more software.

Now, as comfortable as this remedy sounds, it does have one drawback: it doesn’t work.  Just like with the body before it, we are largely failing to view networks as interconnected systems of purposeful parts.  Because of this, our ability to understand potential failure modes, our ability to anticipate, plan for, and prevent them, is compromised.

This is especially significant in the realm of SCADA systems and process control, because there are fewer allowable failure modes and the complexity of the network is higher.  In IT, if a server goes down, people are inconvenienced, but they have intelligence and agency that allows them to compensate and still get things done, albeit at a lower rate.  On SCADA and process control networks, the users are PLCs, sensors, and microcontrollers, and they can work without the network no more than you could pick up a cup of coffee without nerves between your eyes, brain and muscles.

In traditional control environments, there is an understanding of the failure modes of the components of the system and the way these failures impact the rest of the system, and there are entropic fault rates that have allowed us to make accurate estimates of system reliability.  But now, control has moved to a networked environment, and this has created new complexities that aren’t fully understood, and has introduced new, non-entropic fault types that we don’t know how to model.

Obviously, we need to learn how these new fault types affect system reliability, and we need to understand how they are caused and how to mitigate the risks.  And of course, this has to be done while accounting for the stringent availability requirements that differentiate ICS from IT.

The key to this is knowledge.  If we know what produces data, where it travels and what consumes it, we can deepen our understanding of how the process will be disrupted if any part of it fails.  If we know how these components behave and where their weaknesses are, we can deepen our understanding of which disruptions are most likely and choose our compensation techniques to maximize their effect on reliability and minimize their scope and cost.

Asset management and change management are important parts of this equation, because effective judgement needs accurate knowledge.  For the same reason, knowing about device vulnerabilities is important, as is understanding what part of the process the devices drive, and how they work together.  With all of this knowledge, we could conceivably have the level of certainty in networked environments as we do in normal entropic-fault environments, but the sheer volume of information involved means we’re just as likely to choke on it and fail to identify cyber risks, which would leave us right where we are now.

At Wurldtech, we are working to automate the analysis of this information.  Resilience profiles for devices are part of the equation, allowing us to gather everything we know about a device into a concise summary, but we are also extending this by developing techniques for threat modeling network-based processes which will further condense this information and allow us to predict the nature and severity of failures, analytically determine the key points of risk to a network, and find the most cost-effective strategies to deal with them.

- Reid Orsten

Design for test - a fundamental tenet in the development of quality software. [“Quality software” satisfies its requirements and ONLY its requirements, and runs in acceptable time and space.] Given this basic tenet one would imagine the Developer works closely with his test counterpart, collectively specifying and designing efficient tests to assure an artifact’s quality. Of course, one would also expect the converse, the Tester and his development counterpart working in concert to define which test result data is most useful to quality improvement and process evolution.

Sadly this far from the case and I didn’t give it much thought  until I attempted to “strengthen” the developer-tester cohesion in my own company. 

We talk about how improved PCN security is impeded by cultural differences b/w operations and IT folks, well, funny enough, the same can be said about software quality; its improvement is impeded by cultural differences between testers and developers.

Consider the following:

• Developers write code for requirements that are satisfiable and in the case when they’re not, the requirements are “adjusted” accordingly.
• Testers write code for requirements which, by their very definition (to prove the quality of an artifact), are not satisfiable and are excited by such.
• Developers expect their questions have answers, testers accept theirs do not.
• Developers, in general, work, live, and play in the complexity space of P - problems solvable in polynomial time and space.
• Testers on the other hand, live, work and play in NP – problems only solvable in exponential time and space (effectively computationally intractable, i.e. problems for which there are no feasible solutions).  Approximations, probability based randomized algorithms, best guess algorithms, are common place.

These may seem like minor differences, P, NP, etc. But they are a fundamental indication of an overall mindset that, generally, mixes like water and oil: the best you can expect is a weak emulsification.

The Developers and Testers who embody their profession (yes, the dedicated zealots such as I) are diametrically opposed ergo they yearn to work further apart, ergo neither job is performed optimally, hence design for test is handled extremely poorly in practice . And that’s reason #4,125 commercial software is exploitable.

Bryan Singer, VP of Security Services for Wurldtech, is heading to Washington DC and Capitol Hill next week, 20-22 May, as part of a delegation from ISA called “The Automation Federation.” The purpose of this group includes several facets, but Mr. Singer is on one of two teams selected to represent the needs of cyber security in industrial settings and critical infrastructure. The event includes meeting a number of US Senators and Congressmen, as well as a number of other meetings and briefings on Capitol Hill and at the White House. This would not have been possible without the strong commitment from ISA and the hard work they have put into turning this vision into reality, and the meetings would not be possible without the gracious support of the members of this group and their companies.
Here is the Press Release

This is a very important event for industrial security and seeking greater recognition and support in addressing the challenges that face critical infrastructure and all industry. Mr. Singer is extremely honored to be able to participate in this important event and all of us at Wurldtech and across industry look forward to the results that the members of this federation are sure to achieve!

There’s been a lot of activity in the past 6 months in various blogs about the need for responsible public disclosure policies for members of the ICS security community.  While the IT community has developed accepted policies and procedures over the last decade, vulnerability disclosure is still recent in the ICS security community, and it is only starting to grapple (as the IT community first did) with the issues of disclosure.  Unfortunately, the differing natures of the critical infrastructure, SCADA and process control industries compared to the IT communities preclude adoption of their relatively mature disclosure mechanisms and policies, and we must develop our own.

But it’s not enough to develop and publish a disclosure policy.  It must also be effective, actionable, understandable, and meet the needs of the industrial automation community. And development of a good policy starts with addressing the W5 questions about disclosure.

Who do I disclose to?
Vulnerabilities need to be disclosed first to vendors, then to the industrial cyber security community, then to the asset owners and operators of critical infrastructure, and finally to the public.

Vendors must take first responsibility to come up with approved patches or workarounds, and to notify their customers where possible on what to do about it.  The industrial cyber security community needs to know early on in order to develop and deploy effective design strategies and security programs to mitigate their customers’ risk.  Asset owners who employ vulnerable devices in their operations need to be informed of the vulnerabilities to understand the effects on operational risk, and how to reduce that risk.  And only then, once the vulnerability has been characterized, patches produced, and strategies employed to mitigate the risk in critical infrastructure operations, should a disclosure be publicly disclosed by a coordinating center such as US-CERT or CERT/CC.

What information do I disclose?
Vendors require as much information as possible that will help them to reproduce the fault associated with the vulnerability, and diagnose the cause. The ICS security community needs to know the effect of the vulnerability, and how it is triggered, plus any mitigation strategies that are known.  Asset owners and operators need to know the effects of the vulnerability and how it impacts the reliability of their operations, and mitigation strategies to reduce or remove the risk associated with the vulnerability.

When should I disclose?
You should disclose the vulnerability as soon as discovered to the device vendor.  It also makes sense to disclose to members of the ICS security community very soon afterward, as it often takes considerable time to produce a patch or workaround, and the ICS security community is in the best position to develop and deploy interim mitigation strategies to quickly reduce the vulnerability’s impact on critical infrastructure operations.  Asset owners should be notified once effective mitigation strategies have been developed, whether interim or permanent. And only once the patches and mitigations have been fully propagated to the ICS community should the vulnerability be publicly disclosed through disclosure bodies such as US-CERT or CERT/CC.

Where should I disclose?
Disclosure should not take place in public forums, as there is little or no control over vulnerability information propagation and use. Instead, disclosure should take place through secure traceable channels of communication with identified authenticated individuals, using appropriate non-disclosure agreements in place restricting the uses as well as the sharing of sensitive information.

Why should I disclose?
It is the responsibility of the industrial control and critical infrastructure communities to maintain the trust of the public with respect to the reliability of operations.  Vulnerabilities affect all of us, and we need to share information that will allow us to reduce both individual and collective risk on the reliable operations of critical infrastructure. Keeping knowledge of a vulnerability a secret is both short sited and does a disservice to us all.

By disclosing vulnerabilities in a responsible manner to the industrial automation community, we ensure that the development and deployment of patches, workarounds, and other remedies take place in an effective manner to the benefit of all.

From Policy to Action
It’s still not sufficient to craft a policy and put it up on your website for all to see. To be truly effective, your vulnerability disclosure policy must become a core value of your organization, and every employee in the company must believe in timportance of adherence to its principles. Only then will it translate into the everyday activities of your business.

To view Wurldtech’s Vulnerability Disclosure Policy, please click here: http://wurldtech.com/legal/disclosure_policy.php

- Breen Liblong

�

General Notice

We at Wurldtech Security Technologies continue to discuss ways to provide and collate the most useful, relevant, and industry driven perspectives on the issues of industrial cyber security. As such, we have implemented some changes to our blog. We certainly appreciate each of our readers continued support and insight! Considering the comments, links, and re-posts that we’ve received, we know that others find this material useful in their own practices.

A couple of noteworthy changes:

• Blog Content: I (Bryan Singer) will continue to write on industry perspectives, current issues, think pieces and positional arguments on challenges facing this industry, and key events that are all relevant to issues faced today. Dr. Nate Kube will write on the value, benefit, features, ROI, and other topics related to device testing. Dr. Kube is one of the most well recognized premier talents in formal test methodologies and testing strategies for industrial components. His insight and experience continues to offer key value and strategies for uncovering issues and optimizing time to market for vendors, asset owners, and any others seeking to improve the reliability and resiliency of their industrial environment. Frank Markus will write on penetration testing, vulnerability discovery, security assessment, and other key technical issues on industrial security. Steve Kim will also drive key industry news, announcements, and other useful topics. Other authors will occasionally drop by to add their thoughts as well. With this team, we are confident that this blog will continue to drive very useful content to market!
• Comments Allowed: Part of the value in forums such as this comes from interaction with readers. Originally, this blog was more of an “announcements” page, but that has grown over time. In the next phase of this significant effort, we openly invite and ask that any and all readers submit their views, questions, and other comments. We continue to maintain some of the highest standards in the industry for responsible disclosure, and as such we will require free registration and moderated comments to ensure that no sensitive information is accidentally disclosed. Register, read, and add in your perspectives and questions today!

Thanks to all the supporters and people in the SCADA, process control and critical infrastructure industries that read and comment. Healthy discourse and discussion only serves to assist industry in the continued challenges and opportunities in industrial security!

There has been a lot of talk out there recently around FUD and public disclosure, with the usual industrial cyber security pundits weighing in on all sides. With videos such as the Aurora demonstration, Ira Winkler’s RSA presentation, and a whole host of other inflammatory articles, people are wondering whether such articles are doing any good or not.

I have heard some great analogies lately as well. Notably my favorite came from Jake Brodsky during a podcast interview he and I did together over at Digital Bond’s website with Dale Peterson. He likened such articles to an “Energy Drink.” Great analogy: You get a big buzz and lots of energy, and then an even greater crash. That seems to be what happens every time an article like this comes out.

I have my own analogy, and I must pay tribute to an old confidant since my early years, Col R. B. Thieme. Regrettably one of my longest known advisers is succumbing to Alzheimer’s disease at nearly 90 years old, so it seems fitting that I should borrow from such a person. The Charge of the Mosquitoes: while one is not enough to get you, over time they eventually overwhelm you. This is what I fear such articles are doing to the state of industrial cyber security. While there may be some short lived awareness benefit, the greater fear to me is that we are desensitizing ourselves to the issue. In absence of PUBLIC events, every time something like this comes out, and we still have not seen an incident, people believe less and less, to the jeopardy of the whole issue. It’s what I and many others call the “September 10th” syndrome… the fact that few would have imagined the possibility of 9/11 until it happened.

This is a product of human psychology, known as “Cognitive Dissonance Bias” in which our brains tend to automatically reject possible outcomes based upon biases formed by our own experiential base. It is especially powerful with NEGATIVE analysis. For example: Go to a gem field…Turn over five rocks and find five gems, you will likely think you are very “lucky” but not tend to believe that it will happen under any of the next five rocks. On the other hand, turn over five rocks and find no gems, you will likely conclude that you will not find any gems at all. The problem is that this has absolutely nothing to do whether or not there are any gems under any other rocks… Absence of previous data is no indicator of future absence as well, as in security. Just because there is no evidence of past events doesn’t mean it won’t ever happen. Quite likely the opposite is true, but own cognitive dissonance bias drives us to believe that nothing will happen. Yes I know this is a simplification, but the intent of this article isn’t to go through the various ways that people’s minds can be persuaded to perceive success versus failure.

Hearing things like these inflammatory articles only helps to encourage the process. It makes people reach conclusions of “this can’t happen to us” even more quickly by steeling their minds over time, to the point that eventually they will discount it completely. I’m reminded of a story of an old mountain man near Mt St Helens before it blew. Geologists and experts warned everyone to leave, but he was actually interviewed as saying that because nothing had ever happened in all his years, nothing was going to happen. Apparently he never learned that geology doesn’t change on the scale of human lives. Talk about a misstep, he is among those never accounted for. While awareness is good, FUD and inflammatory articles are not, it is making the challenge of implementing cyber security even more difficult. In Part 2 of this article, I intend to address some ways I think we need to change the message.

We are proud to showcase some coverage on Dr. Nate Kube’s two presentations at the Yokogawa Technology Fair & User Conference 2008 in Houston, TX.

Courtesy of Control Global, here are the links to recaps of his talks:

Talk 1: Best Practices in Securing Process Automation Networks (April 9, 2008): http://www.controlglobal.com/articles/2008/129.html

Talk 2: To Fix System Bugs, Wurldtech Gets Fuzzy Wid It (April 10, 2008): http://www.controlglobal.com/articles/2008/132.html

Enjoy.

Yesterday (April 10, 2008), the Sound OFF! blog mentioned an article, “Industrial Control Systems Killed Once and Will Again, Experts Warn”, written by Ryan Singel from the WIRED Blog Network (April 9, 2008).  Unfortunately, a past cyber incident has now been linked to a fatality.  The incident in question is the tragic rupture of a pipline which spilled 237,000 gallons of gasoline into two creeks near Bellingham, Washington.  The gas ignited and killed three, injuring eight others. 

A recent re-examination by security experts into this tragedy has revealed this incident was due to a control system computer issue.  The finding brings new impetus for the need for cyber-security standards and security policy for governments, businesses and critical infrastructure organizations…

To read the article in its entirety, please visit: http://blog.wired.com/27bstroke6/2008/04/industrial-cont.html

To see this post on Sound OFF!, please visit: http://www.controlglobal.com/soundoff/?p=2571