|
Automated Testing of SCADA Protocols
VII. COMMUNICATING THE RESULTS
Since testing cannot guarantee device correctness, testers must be careful about interpreting test results. Vendors and asset owners frequently ask us questions such as:
- My device passed your tests; what does that mean?
- My device was vulnerable to certain attacks; can it be protected?
In principle, there are two main ways to answer these questions:
- We can point to other devices which passed the tests and had good field experience.
- We can explain the tests and their results.
In practice, item (1) is extremely difficult to accomplish for device testing. It is hard to find multiple fielded devices which have all passed the same test suites. Further, data on field errors is rarely shared, for obvious reasons.
Item (2) is primarily a communication, as opposed to a technical, problem. Explaining automated tests is difficult. Thorough device testing often involves dozens of test types and thousands of tests. We see three main approaches:
- A carefully designed taxonomy is essential. A detailed description of all the test inputs and logs is far too cumbersome for communication. Tests could be organized by protocol layer or by type, e.g., resource starvation tests versus protocol robustness tests. Many tests will be parameterized. For example a DoS attack on an Ethernet port might be parameterized by a range of frame rates and frame lengths. Test descriptions must indicate the parameter values.
- Many devices will be vulnerable to some network attacks, especially DoS attacks at high rates. The test report should provide the basis for mitigation efforts. In particular, the test report should indicate clearly where off-the-shelf traffic shaping and packet filtering technologies will be effective.
- Standardization of test types is critical. There is already some agreement but also a great deal of misunderstanding. In the device testing domain, there is nothing like RFC 2544 which has widely influenced Ethernet switch tests. A device testing standard may well emerge in the form of a widely used executable test suite.
« Prev | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | Next »
|
