11 January, 2007

Vista Conversation

Dale linked to D-Bunkers post on Vista and potential ramifications of DRM requirements therein.

Control Engineers need to both be aware of the need to patch and update and have an understanding of when it is relatively urgent. There is no real reason not to patch your systems. To go a step further control vendors need to start developing a organized and controlled mechanism for updating and patching the historians, MES and even PLC's. As cycle time shortens vendors that have already developed this capability will come out ahead.

Obviously all of this has to be done with proper change management.

On the flip side it is absolutely essential that companies like Microsoft and others realize that as they expand more and more into the automated control world they need to have a greater sensitivity to allowing the customer to control when, how and where any changes of any type occur on systems. If they cannot achieve this with their standard deployments then they need to develop deployments that are able to do it.

I'll go one more step further. If you are an engineer and a new system that a vendor is pushing you towards runs an application or OS that performs updates and changes without your full control, you have an obligation to NOT use that system. This is exactly what D was describing in his post on Vista and DRM.

If Vista performs updates and takes actions without allowing the administrator to control those actions in terms of when, how much and even if they occur then Vista should never be used in any closed or open loop control environments. Period.

With root kits and other driver level attacks becoming more prevalent it is good for MS to protect the drivers and ensure they are not the bad guy, but for process control systems they need to do so in a manner that leaves complete control of the process in the hands of the owner of the system not in the hands of some arbitrary algorithm. I don't believe this is some greedy driver licensing scheme (though I could be wrong).

In the SCADA world the ability of the opertor and engineer to fully control the operation of equipment trumps all.


Jake Brodsky said...

Hmmm. I'm not as patch happy as you seem to imagine that I should be. One comment I heard recently came from someone with impeccable credentials: He pointed out that more damage has been done by ill-advised patching than by actual attacks.

We need to patch; but more than that, we need to patch carefully. The problem is testing these patches before we deploy. Many vendors do not give much information about what patches are safe and which ones aren't. It's simply too difficult to do that much regression testing.

The solution is isolation. Real Isolation. Most IT types can't imagine being disconnected from the Internet, but that's exactly what I'm suggesting. The actual amount of information one needs to pass in real time from a SCADA or DCS system is usually quite minimal.

I've stated this before: Many if not most SCADA systems could fit the real time information one needs to pass on a clipboard.

Most SCADA and DCS systems, however, have been connected to the corporate intranet because people thought there was information there that they could use. Sadly, most do not understand the data or its limitations. The IT Division doesn't know this, though. They believe that it MUST be connected because everything else is.

SCADA and DCS systems do not need to be connected to anything. The summaries can be sent to the office. That's how we effect security. It works. We don't need IT. We need sanity and practicality among those who are supposed to set up the information flows. The patches can wait...

Jim C said...

Sorry Jake I know you have a more conservative take. I am the patch happy one. The real focus for me was trying to stress that engineers and operators have a responsability to reject products that automatically initiate actions without the involvement of people (specifically operators).

On the Patching side I tend to follow Paul Doreys take (at BP). They are being connected wether we like it or not so they need to be patched and can be patched. Of course it has to be done carefuly and properly hence my change managment comment.

I spent several years at a manufacturing company trying to keep the networks completely seperate. At the time I was on the IT side fighting the same battle you are but from a different angle. I finally gave up.

Ron Southworth said...

Hi Jim and Jake,

I would love to have our systems here stay as Islands of info. Our management is fairly insistant that we move our control systems into a mode where they can be operated via a mobile environment and continue to have long period's of time where no one even looks at the screens.

The only time we will operate in a "stand alone" mode is when the threat level rises to L4 and above here. As we know things are pretty quiet here on that front (thankfully).

Patch and change management is a tricky subject. I have seen that unless you are on the high end of the market the resources for providing customers the level of in depth support for patch reviewing and their impacts is going to be minimal. This means that the end user has to perform this role of in depth testing of patches etc before deployment into the "production" environment. The net result is more staff will be required to fill in this gap that would otherwise be outsourced. An opportunity for the vendors lost!

I would love to disconnect all the external links save the field device communications and be operating in the mode you have estalished there Jake but at the moment there is a lot of resistance to this. The culture is still centred around minimising costs and not best practices. The clock is running to the next event and at the end of the day if they don't implement all the best practice methods another event will occurr.

Here we are going to establish a test bed environment for system performance testing, change management and patch validation and hopefully some form of off line operator training enviroment as well. There is some simulation software for DNP3 at least you can use to simulate your site communications available in Aus that is afordable. Solutions like this are part of the means to keep the change management monkey off our backs.

We are suffering a lot from connectivity for the sake of laziness at the moment but this I am certain now will change after our external auditors come in and assess the system. The good news fro me is the people that will be coming along they have a process control background and this seems to be of some concern to the IT staff. I wonder why...


CNI operator said...


Seems to me that the systems you describe have external connectivity just for the hell of it!
In many of the systems we run, we derive substantial value by exporting data from control systems to external systems including trading systems (by external I don’t mean Internet, I mean Intranet). These data flows can be relatively large.
Reverting to an isolation model is an option we only take in an emergency situation.

The trick, of course, is how to make the data export (and import) as secure as possible. Or at least secure enough wrt the risk.

I don't subscribe to the view that IT is bad, Control Engineer good. My experience is that IT folks bring a lot to the table BUT we control folks have to educate them.
Sure they will want to use the tried and trusted methods they know from the IT environment, it’s up to us to explain why these methods are not suitable for control systems.
BTW, the more I have these types of conversations with the IT folks, the more I discover that standard IT security solutions have a place in the control world.