Data Link – Layer 2

Second Part of Layers Physical Security – Layer 1

OK - Second Layer. In OSI it is the Data link layer with Collision Detect, Collision Prevent, Ethernet, Token, TDM, and all of the others. In a nutshell it is how the systems talk to each other on a point to point basis. When you are talking Ethernet and switching (especially spanning tree) you get overlap into layer 3.

There are a number of areas where it is significant from an information security standpoint for SCADA systems. In the last 10 years the conversation has become dominated by the Ethernet issues but there are other significant issues occurring as well particularly in the wireless realm.

RS-232 (now iea232) was the prominent linking mechanism for quite some time (defined in 1969). The PLC’s can play the part of either DTE or DCE depending on its function in the design. There are some huge advantages to the RS-232 usages. It supports deterministic timing meaning that actions and responses can be watched real time and reactions can be based on ladder logic layouts without much concern of a “lost” packet. It supports a sufficiently high data transfer rate for most automation processes and it has been well tested and used. RS-232 is falling somewhat out of favor as a connection mechanism in the automation world and largely being replaced by Ethernet for local connections. (boy that sentence is going to generate some hate mail) Although IP is really at the next layer it is part of this shift and in the rest of the networking world this shift happened over a decade ago. If you look at my older posts this synchs with my stand that the automation world lags the rest of the information systems cycle by two to three generations and 8 to 10 years.

There are some substantial security implications of this shift to Ethernet. First of all the shift has just started. Less than 20% of PCN’s are Ethernet but most of them (say 80 to 90%) have direct control connections to the Ethernet network via various aggregation tools/methods such as RSLinx. Ethernet, while very reliable if properly deployed, is definitely not deterministic. Multiple nodes exist on the same structure and they work on a modified collision detect structure. If one node is talking the others wait random periods of time to start. Switches mitigate a lot of this by separating the collision domains but when a destination node is receiving traffic from multiple sources there are still lost packets. This is largely overshadowed by significantly greater data transfer rates.

There are some very specific weaknesses to Ethernet that I am concerned about in the PCN world. The most prominent is ARP spoofing. Without getting into the details (I’ll save that for the follow on PDF’s I am starting to write) arp spoofing involves taking advantage of the way ethernet makes connections to allow one node to “pretend” it is another node. Although I have never seen personally, or even heard of a case of arp spoofing on a PCN the entire architecture would be very vulnerable to it. I think the biggest reason it hasn’t emerged yet is that there is no real need to do it at this point. If there is no authentication to a MODBUS IP node anyway why bother pretending you are from somewhere else. As ACL’s and in line firewalls increase in the prevalence I think the frequency of ARP attacks will increase. This could have a very significant impact on devices that are so fragile that they croak when a syn scan hits them.

Controls for the Data Link Layer are pretty simple. For Ethernet MAC filters (Recently in the form of NAC) and switch configuration shutdown of ports (which overlaps with Physical security) serve as a first layer.

NAC is emerging but still needs some development. What it really comes down to for NAC is that a device needs to talk to an end point to be authenticated in any way (let alone a fancy key exchange followed by certificate verification). Since it needs to talk it has to be given the opportunity to connect to the network. What this eventually evolves into is a means of quarantining a device in an “unauthenticated” VLAN until it is verified by some means. This inserts multiple points of opportunity to overcome the defenses. Any time the layers work against security instead of for it you can almost guarantee that someone will find a hole.

The NAC schemes that seem to be most likely to succeed involve Identification of the MAC as an accepted MAC by an authentication and verification that occurs in a quarantine VLAN. A lot of the schemes are using DHCP because it already has a means of differentiating based on MAC address but this has the weakness of not covering for static addresses. All of these NAC methods require upgrades or replacement of existing hardware for most implementations. Other NAC schemes involve searching for the bad guys and using some other mechanisms to expel them from the network.

VLANS are the next major control associated with layer 2 in the Ethernet environment. Basically they are a means of segmenting traffic into separate “networks” on the same devices. They can be set up using different mechanisms as the differentiators for which traffic belongs to which VLAN. The most common I have seen at sites is a simple port assignment. With this mechanism ports 1-6 are assigned to VLAN Bob, ports 7-12 to Alice and so on. Since each VLAN is a separate logical network they typically “cannot” talk to each other without a layer 3 connection. VLAN’s are often associated with a specific IP subnet (sorry layer three + again here).

The last part is the catch from a security perspective. Network Administrators and Engineers almost always assign a gateway for each that has no filter or ACL to prevent Bob from talking to Alice or worse yet Eve (at a completely different site) from mugging Bob. Just because they are on a different subnet does not mean they cannot talk to or interfere with each other. The problems for this don’t occur at layer 2 but when designing, operating or auditing you shouldn’t think that being on a different VLAN by itself is a protection. A further complication with VLANS set up via Port assignment is that there is often a VLAN used for management or troubleshooting that is assigned the entire port range (or at least overlaps other ranges). Any device that bridges these also serves as an entry point. It also serves to complicate the design.

VLANS can also be based on source MAC addresses, QoS classifications, IP addresses and other means but I haven’t seen that much of the more detailed assignment mechanisms in the ACS world. MAC address differentiators are sometimes used but have most of the same pitfalls of the port based VLANing. Some realistic NAC implementation mechanisms try to take advantage of MAC based VLANing to provide the quarantining I mentioned above.

Key point here. Just because it is on a different VLAN does not mean it is segregated. Try something simple. Ping it from one device to the other.

Already too much writing for the weekend. I’ll continue later this week.

Update:
Continued Here

Digg This

Comments - Wireless Conserns and Realitive Risk

Ron has a good comment up on the Physical security post. It deals mainly with various aspects of the wireless discussion and the use of reserved vs. common bandwidths. It also touches on something I should have expounded on better.

Any of the items I mentioned might be too extreme or not extreme enough depending on what the relative risk level is for a given system. For one system (on the physical protection level) an electrified fence with constantine wire, an armed guard and trained guard dog may be appropriate. For another a padlocked plywood door might do.

One of my key points from the post was that certain design factors of the physical system can be considered mitigating factors for information security risks related to the ACS.

Alex at Riskanalys deals very well with mechanisms of determining how significant potential issues can be. Hopefully some time in the future he an I might be able identify subsections of impact modifiers, threats and controls specific to DCS and SCADA.

Physical Security - Layer 1

As I have said previously I tend to think of Security layers in terms of an expanded OSI model. This might be somewhat simplistic but it does provide an easy structure for a working defense in depth strategy. In many cases it also matches well to the domains, objectives and ISO categories. In areas where it deviates it often fills gaps rather than creating superfluous work.

Strictly speaking layer 1 deals with the standards for physical connections radio and wireless characteristics and timing and signaling mechanisms. I am not talking about the actual OSI layer I am just using it as a conceptual guideline.

Physical Security is one of the fundamental pieces of the information security structure and is essential for proper defense in depth. Physical Security requirements are recognized in ISO 17799 as a category, within CoBiT in multiple control objectives and in ISC2 as a domain. It is often one of the more difficult aspects to deal with. Direct control of Physical Security is often out of the hands of IT or Engineering (typically for good reasons). Wireless mechanisms complicate proper implementation of physical security by bypassing existing mechanisms of control. Finally many Physical Security best practices and needs fall outside of the actual scope of Data Security. All of these are standard complicating factors when dealing with Physical Security.

Within the Automated Control world the physical security becomes far more complicated in that it also includes aspects of safety. While many of these are issues that properly reside in the responsibility realm of the engineers and operators it is still essential that the people responsible for managing information security risk understand how they work. Though they are not directly part of the information security realm, often proper physical security and physical design parameters can mitigate much or even all of the risks presented by information systems ties. There are also some unique challenges to obtaining the typical requirements for physical security of information systems.

Perimeter Security, Controlled access, Manned monitoring and reception, Environmental Controls, Control of access to cables, Public Areas, Secure Disposal methods and Monitoring of support infrastructure fall within this realm in typical Information Security implementations. Within ACS deployments Fail Safes, interlocks, inherent physical characteristics, proper finite element analysis and redundant essential systems (three pumps) greatly reduce risk of issues in critical systems. These should be added to the standard list of physical concerns to understand for information security professionals that deal with SCADA systems. When properly implemented, these design criteria and mechanisms can alleviate many of the concerns that are often cited in information security risk profiles for SCADA or ACS.

Perimeter Security is the establishment of a clearly defined boundary with controls to ensure that only the proper people have access to the equipment and systems within. The typical perimeters are walls, fences, hedges, cages, and separate offices or buildings. To be effective they have to be combined with controlled access and manned monitoring. Wireless systems circumvent perimeter security mechanisms completely and therefore must have a differentiated access control mechanism instead. ACS and SCADA complications to perimeter security mainly deal with scale. Some oil fields span hundreds of square miles, Power Lines are ubiquitous and have many unmanned transformer and switching stations, water systems and pipelines go through towns, cities and neighborhoods and can stretch for thousands of miles. While remote pumping and transformer stations usually have perimeters they are rarely manned. For reasons that have nothing to do with IT security they are usually well monitored in the form of alarm systems and physical access barriers but often the incoming telecommunication systems are accessible outside of this perimeter. A mitigating factor to physical access risk that deviates from a standard IT environment is that many of these systems are so remote that it would be very difficult for someone who is not already "inside" to access them. The North Slope and offshore rigs come to mind. This mitigating factor should be considered but not always relied on.

Controlled access includes locks, gates, key card entries, and the reception lobbies. For wireless systems it includes the authentication mechanisms. All of the encryption in the world is useless if you have no means of authenticating access to the root system. This was the entire nature of the misunderstanding of WEP for 802.11 and all the problems that have stemmed from those mistakes. This same gross conceptual error also extends to the spread spectrum systems being deployed currently in many SCADA and PCN environments. Just because I am unable to intercept communications between a base station and a node does not mean that I cannot connect to that base station directly provided I have the right settings. Without some form of authentication it becomes a function of security by obscurity. All of the devices and networks become accessible (sometimes from up to 100 K away) with one mistake.

Eventually any physical barrier or controlled access mechanism can be bypassed. At this point manned monitoring becomes an essential piece of the physical controls. Typical monitoring mechanisms are direct manning, patrols, cameras, log reviews and equipment monitoring. The last piece is one of the greatest mitigating factors for good ACS security. Almost all operating machinery has an operator somewhere monitoring it or the system attached to it. By properly using/training these individuals a significant reduction of risk can be obtained. The presence of these operators is one of the significant advantages that many SCADA environments have over the typical office environment. In some other post I will discuss Segregation of Duties and how in many cases these operators are one of the most likely risks but for the purposes of enhancing physical security they are one of your best assets.

Interestingly enough Environmental systems are often one of the stealth ACS environments out there that almost every organization is dependant on. HVAC systems are essential for the proper operation of any data center and are more and more likely to be controlled by network accessible interfaces. It is also becoming increasingly common for power distribution panels to have standardized Ethernet accessible PLC's controlling them. Other than the realization that these systems are increasingly likely to be able to be hacked there is little to differentiate the physical environmental requirements of ACS vs. Standard IT systems. Redundant power, proper cooling and heating are all important. One thing for engineers to keep in mind is that many security systems such as firewalls, NIPS and switches are designed for a data center environment. They may not perform well in a shed that reaches 20 below zero. I have seen a firewall implementation mandated by information security have difficulties with MTBF for precisely this reason. Note to vendors - If you want to get into the SCADA market start designing more resilient equipment. A typical Ethernet switch placed 10 feet away from an operating paper machine rarely lasts long.

Control of access to cables can be very problematic in a PCN environment. When a network extends for miles there are any number of points where access can be obtained. Fortunately there is some mitigation in the form of departure from typical Ethernet connections (at least as long as it lasts). Most extended networks require some form of longer range layer two connectivity's. I will discuss these items somewhat in layer 2. Including the fiber runs within trenches or other relatively inaccessible paths can help further mitigate risks associated with this control but for large geographic areas there are definitely challenges. For facilities with defined areas it is worth ensuring that cables that cross public roads or areas are not easily accessible or are protected at another layer if it is unavoidable. A key problem I have seen with this is RJ-45 outlets to a PCN Ethernet segment without any identification of the network type or any way of controlling who plugs into it. This often occurs when an engineer thinks it is alright to put a PCN connection in a conference room (or office, or even home) that he commonly uses. While not absolutely essential complete physical separation (including switching infrastructure) of PCN from all other networks should be considered. If the system is safety essential, critical or "red line" such as ESD systems then complete physical separation should be considered essential.

For the IT people reading "Fail Safes" is the failure mode of specific equipment or systems. As an example valves fail in three modes, Open, Shut, or as is, with a loss of power. The engineers who design the system determine which failure mode provide the most safe environment for a given system and status. Interlocks ensure that when certain devices or systems are operating in a specific manner that other specific actions cannot happen.

From an information security standpoint an important aspect to consider is the dependence off failure modes and interlocks on programmable controllers. Ideally A fail safe position is a fail safe position and nothing can alter it. It is an inherent part of the system. The same should be true for interlock responses. The problem occurs usually when specific programmable settings are used to enact the fail safe or interlock and those settings can be altered. I have seen some problems with this in some ladder logic deployments (essentially a series of inter dependant switch positions). Because controllers are more likely to be remotely configurable it is more common to see interlock settings and fail safes that can be alterable without the knowledge of the operators or engineers. This is one reason that control of physical access to the PCN (and by extension the PLC's) is so important. The flip side of all of this is that if the fail safes, interlocks and other inherent design considerations are done well it is very difficult for any failure mode to cause any significant issues. In a well designed system three or more sequential failures (at least one of which should be a physical property of the system) must occur before safety is compromised.

I couldn't tell you how many times I have sat in a room with an Information Security professional talking with Engineers and the IT guy states that one of the risks include fires or explosions. The engineers usually just roll their eyes. The fact of the matter is that in a well designed system even if an operator with complete access to the systems forcibly does things wrong it is usually very difficult to force a catastrophic failure. Of course I have also seen the reverse of this happen. If the failsafe is dependant on the proper operation of a PLC and that PLC configuration becomes suspect then that failsafe is no longer dependable. When an engineer learns of this the response is often a great deal of concern.

Digg this story

Measurable Layers

Prediction #9 seems to have gotten at least some attention. I have had three separate requests (one in two parts so I am certain the author is interested) for expansion, and clarification.

I guess this is good. I obviously tapped into a healthy meme seed but I do have a bit of a dilemma.

I am not really sure what I meant.

Well I am sure what I meant but I am not really sure how to articulate it. (wow doesn't that leave me an easy out in 08)

Every attempt I have made turns out to just be a small part of the whole. It is like trying to draw a hypercube on a piece of paper. All I wind up with is a bunch of weird looking triangles, rectangles and squares.

The way I used to look at security was as a sort of modified OSI model. (way back when)

Control Physical Access
Locked TC and DC doors, Building Access, Wireless Access Controls

Control Switching/Electrical Access
More Wireless access controls, Mac Filtering, NAC (if it ever works), VLAN’s

Control Routed Access
ACL’s, Good Subnetting (Yes I know a subnet doesn’t stop anything by itself, but if you don’t get the routing right everything else is harder), Proper DMZ/Extranet/Segmentation

Control of Application Connectivity
Firewalls, Tunnels, Some Proxy Functions,

Control of Sessions and early SoD
Session Segregation, Basic SoD, Identity Controls

Control of Data access and Presentation
Db Controls, Site/share/page access, More Identity Controls, Middle SoD

Application Controls and Control of data manipulation and metadata
Business SoD, Application Design, Business use of Application, More Identity Controls

This approach actually still works in many cases but it lacks a lot of essentials. It is almost purely tactical and has no self awareness. It also focuses too much on access control/preventative controls and not enough on mitigation and prioritization.

A lot of people who talked about the OSI model used to jokingly add a few more layers.

Politics, Religion, and Money

I am not so sure that is a bad idea but I would probably add a few more layers and call them:

Process, Policy, Governance, Compliance, and Money

in that order.

If you do that combined with the other layers it looks a bit like ISO 17799 domains doesn’t it? Well perhaps with some CoBiT Control Objectives thrown in.

There are a few differences though. Instead of interrelated overlapping domains you have sequential (potentially superseding) layers in both directions. These are layers where (for a given threat) you can show a certain level of protection. Multiple layers can be stacked for increasing sequential protection versus a threat from a given vector.

So let’s add these into the mix. Do they overcome the shortcomings? Well not completely. There is one thing still missing, visibility.

So feed visibility as a subset requirement into each of the layers.

As a quick example of that meme:

A firewall is valuable because it stops some attacks

If you are able to see how many attacks occur “outside” the firewall and compare them with how many attacks make it “inside” the firewall you have added value. The value isn’t directly added to the control that is the firewall. The value is added at the Process layer where an evaluation of the effectiveness of the firewall occurs and other controls can be used to mitigate the identified weaknesses. It might also be added at the Compliance layer where an organization might have to meet PCI requirements on proof of effectiveness of controls (specifically the firewall as a Control).

So what I was trying to say when I wrote:

“Vendors that are able to encompass the concept of measurable layers in security will emerge (or in the case of the few that are already out there will do well financially)”

Is that vendors that are able to add or combine either automated or easy to implement means of measuring effectiveness of the controls they peddle will add value.

Also

Vendors that facilitate the process of not only tying controls to specific effectiveness but also representing the effect of overlapping controls on overall risk mitigation will add a great deal of value.

If you can demonstrably add value then you can make money.

That’s what I meant …

Sort of

So now I am circling around to tag the originator of the chain letter.

Wireless, RAS, PPP, Web Gateway and ... Modbus

We have it all

I hope they are doing every piece right because there is a lot of room for error here.

For that matter the vendor might get everything fine and in this design it would be real easy to see a lot of customers mess it up.

They mention the word security "all in total security." once in their marketing blurb but I can't find any white papers at their site.

Show me the meat!!!

No search capability either.

A different kind of Wireless

SciAm has an article on wireless power.

I saw a similar one on the BBC that was linked to by slashdot.

Seems pretty neat.

I wonder if we have any resonant frequencies.

Control Systems

I think it is time to revive one of my pre blog writings trying to get people interested in security issues around SCADA systems. If you want to know what they are and why they matter take the time to read this. It is a big problem and getting bigger. If you are already into the topic it might give you some catch phrases and other angles for the non initiated. A little Campy but I didn't write it for a blog.


Control systems are everywhere. From nuclear plants to elevators, automobile manufacturing robots to remote surgery, multibillion dollar offshore oil facilities to children’s toys, computers are controlling more things every day. Automated control systems are not a new phenomenon. In the first decade of the 1800’s punch cards were used to operate weaving looms running relatively primitive programs using mechanical interlocks and stops to control the equipment. Digital computerized systems have been around for more than a half century and short range radio monitoring and control existed well before wireless became a common term in the IT world. These systems go by many different names, Automated Control Systems (ACS), SCADA, DCS and Process Control Systems are among the most common. Ultimately what they do is what defines them as a separate category. Whatever the name, the defining function of control systems is their ability to directly physically manipulate the real world.

In the last decade there has been a revolution in the automated control world. Mirroring the advancements of the information technology world, ACS have become more integrated, easier to connect to, and standardized.

Many systems are now directly or indirectly connected to an IP network that is ultimately connected to the internet. The key control and programming point of these systems is often run as an application on one of the common Operating Systems.

This standardization and interconnectivity has had a dramatic positive effect on the efficiency, safety and ease of implementation of these systems.

Because these systems are often more complicated than other computing systems, have a higher capital cost than other computing systems, and are tied to physical infrastructure, the adoption of the newest generation lags the IT and internet world by 8 to 10 years. This puts the ACS world right in the middle of the turn of the millennium IT environment. The same paradigms apply. There are and will be dramatic impacts on business models. Irrational exuberance abounds. A huge amount of money is being spent and saved.

Finally the security challenges of the early internet days are now being felt in systems that control our power, water distribution, oil pipelines and wastewater removal plants.

This final point cannot be overstated. The same viruses, spam, pop-ups and botnets that give the IT world and the average home PC user headaches can affect the power supply to your house and business and change the way that the natural gas pipeline in the back of the neighborhood works.

There are two key questions that define the debate about how or even whether to direct resources to protecting ACS. Can control systems be accessed and controlled by unwanted individuals? What will/can happen if they do access them?

The answer the first question is a direct and simple yes. Not only can these systems be accessed but they have been accessed. If a system is connected to any other IT or telecomm system then it can be reached and controlled.

The answer to the second is less direct. It depends. It depends on what the ACS is controlling, how much and how fast a human can get involved and most importantly how the underlying system integrates into the process being controlled. In most cases production can be stopped or efficiency impacted. In some cases people can be hurt or killed, large amounts of environmental harm can occur, and huge amounts of money can be lost.

A number of high profile incidents are easy to find.

The California power grid was compromised and service was almost interrupted, waste water has fouled beaches, David Beckham’s car was unlocked, started and stolen twice, and the slammer worm was found in the systems of a nuclear plant.

From the silly to the terrifying, compromises of automated controls systems are occurring daily. Ultimately these incidents show the public side of the impact but the real threats can be subtle. Control systems are not designed to identify abuse and hacking. Until recently identification of attacks specifically directed at ACS was not available or possible. In many organizations the control systems are not located on a segment of the network that allows easy differentiation of unwanted traffic. The result of these and other weaknesses in existing architectures is that the real level of compromise and therefore the threat and risk levels are difficult or impossible to determine for most organizations without the acquisition of greater information and understanding.

People are doing things to fix it but more needs to be done and faster.

Wireless Vulnerabilities

Some good Stuff over at the new (and prettier) Digital Bond Site.

The new focus on component and device vulnerabilities increases the exposure of DCS devices to threats.

This is emphasised by the recent Broadcom vuln that affects Dell, HP, and Gateway wireless systems.

While it is not specific to this problem (you should still be checking) my last few days of posts focuses on the growing connectivity of PCN's via different types of wireless access.

Sigh - Wireless SCADA

More Wireless Stuff

And

Slashdot

I dealt with "nanosensors" (and the quotes are needed) some in my last job. While small in terms of hardware, they are still in the several pounds range. So they are anything but nano. The ones described in the Slashdot article are smaller but still in development and there are limits to how small a wireless transmitter can be and still be effective at long ranges. (though admittedly the limit is pretty small).

At least in the "nano" sensor story they are only RTU's.

Like I have said earlier

SCADA DCS PCN's ACS whatever you want to call them are essentially at the same point that IT was at in '99 or early 2000.

People know there are problems (and even how to fix them). They are trying to tell people there are problems. Some people say they are fixing them (and a few, very few actually are) but most are marching ahead looking at the pretty pretty baubles.

The wireless fascination has me particularly worried. Some of these systems can be connected to 60 miles away. I realize that many of the technologies being used can be very secure (much more secure than 802.11 can ever be even when properly configured) but the default and improper setup problem will still exist. I don't think that is acceptable for something that is controlling gas and oil pipelines or water and sewage facilities.

It is the same as the euphoria that hit IT in the late '90s

Look what I can do...

We're going to get rich...

Don't worry that will never happen...

Don't point it out it will just make the problem worse...

Sigh

Arrrgghhh - Wireless SCADA

This post is heavily edited from what I originally posted to protect the innocent and the idiots.

Suffice it to say that if you are doing wireless on SCADA systems you should protect more than just the historian. Default settings shouldn't even be possible.

Wireless has been used in RTU type systems for decades in the SCADA world. More recently companies have started to use IP type communications over wireless including Modbus/IP.

There are some good ways to do this but you have to be very careful and very aware of what can happen. Think hyper surgeon generals warning if you are a vendor.

I'll leave this with the close I originally had.

The worry of war driving for Natural Gas Pipeline valve control gives me the willies.

Update:
More Here

Cheerleader Defense

I think it is particularly unwise to base any significant security decisions solely on one court case.

Especially one with the word cheerleader in it.