NERC/CIP Compliance | Achilles Certification Program

No amount of testing can guarantee correct device behavior in the field. Only running all possible tests could do that, and there are generally far too many possible tests to exhaustively run them all. For instance, the cardinality of the test domain typically increases exponentially with the addition of each new variability, a property commonly referred to as combinatorial explosion. Worse, due to timing variations, a device may pass a test once and fail when the test is rerun later. The pioneering software engineer, E.W. Dijkstra, summarizes the situation well: "Program testing can show the presence of bugs but never their absence."

Nonetheless, testing is the most common approach for finding bugs; a failed test definitively proves that a device has a bug. Well-designed tests can also exercise a device in near real-world conditions, demonstrating device capabilities and limitations, qualitatively and quantitatively. Such tests increase confidence in device performance, even though absolute confidence is not achievable. Testing can provide valuable data to support comparisons between devices, including different versions of the same device and devices from different vendors. Finally, testing can provide legal and regulatory protection. As systematic testing of networked devices becomes common engineering practice, it will become increasingly risky to omit.

We now describe the characteristics of Achilles testing in terms of test plan, test methodology, test results, and test automation. An Achilles test plan describes the objectives of the testing, the features of the device-under-test, and the laboratory configuration. The test cases to be executed are described in detail.

Achilles testing covers all relevant layers of the protocol stack. The tests exercise both the protocols used directly by the DUT, e.g., TCP/IP and MODBUS, as well as utility protocols, e.g., ARP and ICMP. The test suite covers the many well-known network attacks, including those based on flooding the communications channel, consuming resources on the DUT, and inserting purposely illegal packet header fields. Since Achilles testing is based on formal methods, quantifiable claims regarding test coverage are possible. Finally, Achilles output checking uses, among others, device monitors; an OPC client continuously captures and evaluates device behavior during test case execution for example.

Achilles testing provides detailed results for each test case enabling the reproduction of the result at a later date. The results also include recommendations regarding device hardening and required switch and firewall configuration.

Finally, when test cases are numerous, manual execution and output checking is expensive, making effective testing unaffordable. With test automation, many tests can be run economically. Tests can also be repeated easily on another device or device version, or when the results on the original device are contested. With Achilles, test automation is pervasive: a test case is a parameterized Achilles plugin and a test suite is a customizable script invoking multiple test cases and monitors. Achilles test suites are easy to execute repeatedly and easy to reconfigure thereby enabling regression testing which is prevalent in any efficient quality assurance cycle.