How to Create Awe-Inspiring Network Protocol Test Suites

Robustness, Security, and Vulnerability Testing

The cost of computer security breaches is no longer hypothetical. According to the Cisco 2018 Annual Cybersecurity Report (page 46), more than half (53 percent) of all attacks resulted in financial damages of more than US$500,000 (for each organization), including, but not limited to, lost revenue, customers, opportunities, and out-of-pocket costs. Interestingly, about 19% of the attacks resulted in financial damages of more than US$2.5 Million per organization!

It is no surprise, then, that product developers have turned their attention away from verifying the correctness and interoperability of their protocol implementations. Most Internet protocols (like TCP or UDP or ARP) have years of field experience, and, with freely available, open source versions, the expectation is that conformance, compliance, and interoperability issues are largely resolved.

Security Testing

Then, as this line of thinking goes, the new focus should be on security testing – verifying that malicious actors cannot breach the firewall, that all phishing attacks are rebuffed, no unapproved programs are launched inside the firewall by bots using zombie systems, no passwords are easily guessed, and no vulnerabilities exist on networked equipment allowing entry to an intruder.

Unfortunately, the burden of protecting against today’s security attacks is shared by the network and system administrators (who are responsible for configuring and monitoring firewalls, and promptly patching vulnerable systems), human users (who can best defend against phishing attacks, bots, and easy passwords), and the developer (who is in the best position to ensure that the app or device is hardened from attacks).

In today’s world, the developer and his test team must address emerging security threats. Fortunately, IWL’s automated network test suites have evolved to address these threats.

“Focus on key areas developers tend to get wrong.”

At the founding of the Company, IWL’s design goal was to focus on the key areas that developers tend to get wrong. This early design goal serves us now as developers use our automated network test suites to secure the app or device.

Beyond conformance, compliance, and interoperability testing, our test suites cover negative testing (subjecting a device to invalid input streams), memory leak testing, stack / buffer overflow testing, and a range of potential app or device-oriented security breaches. In addition, our indepth knowledge of networking protocols means that we can readily identify protocol-specific vulnerabilities.

For example, a software developer may forget to add “unsigned” before integers in the C language with catastrophic results. The app or device will depend on the value of x declared with "char x" ranging from 0 to 255, but instead, those values with range from -128 to 127 (because it should have been declared as "uchar x") and a line of code that reads "if(x > 200)DoSomething();" will not work as intended.
For operational networks, an error of this type will cascade through multiple systems and defy diagnosis as it will be very difficult to find.

Two key limitations of classical conformance and compliance testing are time and expense. If one creates a test for every single “should” and “must” in the standards document, the result will be a very thorough set of tests. However, these tests will require an extremely long time to execute and they will flag all errors, many of which will prove completely inconsequential to field operation.

Having the expertise to differentiate the serious, fatal errors, from small, inconsequential rules, is key to the test creation process and providing a meaningful return on investment for the test organization.

In addition, IWL judiciously avoids the pitfall of creating test software as “an exercise for the reader.” Many organizations offer libraries and partial test implementations, e.g. one or two sample tests, and then leave the rest of the work – creating the tests, documenting the tests, flagging errors, and so on – up to the user. This is rarely a good return on investment for the test organization.

Finally, IWL documents the addition of new tests to our test suites, so that those with the expertise and special requirements can customize the tests to meet their needs.

Follow a systematic process for test creation.

Our white paper, Network Protocol Testing Overview outlines the range of testing required for fielded products operating on a network. When contemplating the creation of tests for a new area, we begin with discussions of the applicability of these test methodologies for the new area. In addition, we search out new approaches that may prove valuable via our work with researchers within the government and universities.

For example, our paper discusses deep path testing, meaning the ability to test multiple paths through code, not just the main path. The new area of concolic testing figures prominently in deep path testing in that it treats program variables as symbolic variables along a specific execution path to more effectively maximize code coverage.

Concolic testing offers the potential of discovering more vulnerabilities in a shorter period of time. Thus, IWL has the potential to generate meaningful test cases with better return on investment.

There are many other aspects to meaningful test creation and the application of a systematic process for doing so that remain proprietary.

Using autonomous systems to detect, evaluate, and patch software vulnerabilities

Our ultimate goal is to support our customers in producing perfect software.

Currently our solutions automatically detect and evaluate software flaws and vulnerabilities in target systems, then generate a detailed report. Software engineers inspect these reports and create patches to address the flaws. In the future, it may be possible to automatically generate a patch for each software vulnerability. At that point, an engineer would have the opportunity to inspect the flaw and corresponding patch and approve or disapprove it.

When we get to that point, we fully expect our customers to be filled with such passion and joy as to perform the software equivalent of bodice-ripping.