The Benefits and Challenges of Cloud Testing for Protocols
As organizations work through the digital transformation process, migrating business applications from their Local Area Network (LAN) to the cloud, they may consider migrating network protocol testing to the cloud as well.
Typically, protocol testing is done on the premises, in a lab, so the activity can be isolated from the rest of the organization’s network in a closed network environment, not attached to the Internet. There are two key reasons.
First, many forms of performance testing as well as protocol testing, introduce malevolent packets and/or high-volume network traffic. If these packet streams were accidentally introduced to the production network, the organization’s business activities would suffer.
Secondly, depending on the size of the organization, the organization’s production network could be an appealing target for hackers and those engaged in corporate espionage. Although both the production network and the test lab network are targets, even more care must be taken that new products (in the test lab network) are safeguarded, as new products are a richer target for bad actors.
So, what are the benefits of cloud testing for protocols?
The Benefits of Cloud Testing for Protocols
1. Reduce Infrastructure Costs
The lab environment requires office space, power, cooling, upgrades, and maintenance. A cloud environment mitigates these investments and frees up space for the organization.
Virtual machines or hypervisors commonly deployed in the cloud environment leverage existing investments in hardware systems, avoiding incremental expenditures.
The cloud environment promises scalability, meaning that if more compute resources are required, the cloud environment can quickly “scale up” without the requirement for IT staff to provision, install, and configure new systems.
2. Effectively Utilize Highly Skilled Staff
The quality assurance staff assigned to testing network protocols are unique; they have specialized knowledge that is best focussed on finding and fixing protocol implementation errors, and not the maintenance of computer systems and networks (the work of a network administrator).
3. Integration with other Test Tools
Network protocol testing applications are typically integrated into “test harnesses,” packet generators, and code check-in tools like SVN and custom scripts for builds.
The test harnesses provide the ability to execute a test suite, save the results automatically, both passes and failures and present those results with other test results in various pre-defined reports suitable for management and the technical staff.
The packet generator receives commands from the test application to send a series of predefined packets at a certain rate to the device or application under test in coordination with each test.
Code check-ins and builds may be integrated into the test harness for custom builds and then deployed to the test application.
Although this integration may be accomplished in the LAN, it is often in “silos,” with each team controlling its own resources. Economies may be achieved by migrating to the cloud with each team cooperating and sharing resources with other teams. (There are some issues, as indicated below.)
The Challenges of Cloud Testing for Protocols
1. Lack of Cooperation
Typically each development and testing team controls its own resources; this is easily done on the LAN. For the organization as a whole, this means that test equipment and test application investment is often duplicated, and many of these test resources are idle most of the time.
However, the organization’s management does not want these teams to fail to meet their schedules because they lacked a test resource. This is the most pervasive argument for retaining protocol testing in the LAN environment.
2. Lack of Deployment Flexibility in the Cloud
The device under test will receive the test traffic from the network protocol test application in the cloud. If the device under test is a software application, it can be deployed on a hypervisor in the cloud. In this case, the automated network protocol testing operates very similarly to the LAN.
However, if the device under test is a hardware product, there is typically no way to place it in the cloud environment adjacent to the test application. The cloud services are simply not set up for this. In this case, the device under test must remain in the LAN. This poses an additional problem in that if the test application is in the cloud and the device is under test on the LAN, then the test results will be affected, in some cases, by network issues with the Internet stemming from the non-repeatable, non-deterministic path the test traffic will travel when reading the device under test.
3. Non-Deterministic Network Path Yields False Failures
The Cloud environment does not offer a pre-defined, deterministic, repeatable path through the network. This is likely to change eventually; the IETF is working on RFC 8655 “DetNet,” a deterministic network that provides:
Minimum and maximum end-to-end latency from source to destination, timely delivery, and bounded jitter (packet delay variation) derived from these constraints.
Packet loss ratio under various assumptions as to the operational states of the nodes and links.
An upper bound on out-of-order packet delivery.
Although the key goal of protocol testing is to verify that the device under test responds correctly to the packets received from the test application, if the network transmission between the source and destination is delayed (which is a reasonable probability on a non-deterministic network path), then it is likely that a test will fail, because the device under test did not respond in time. Some protocols utilize timeouts to determine when to give up on waiting for a response. (Some examples are: TCP, SMTP, and DNS) The test may generate a FAIL or an UNTESTED result simply as a result of the delay between the cloud and the LAN.
Overcoming the Challenges of Cloud Testing
The best way to overcome these cloud testing challenges would be to find a cloud provider that permits the following:
The installation of equipment (specifically, your device under test) co-resident with the cloud infrastructure servers.
The user can have full remote access to your device under test (ability to do reboots, install new software …)
Plan for the implementation of RFC 8665 or equivalent deterministic path inside the cloud infrastructure
Cloud Testing Wrap-Up
Cloud testing of network protocols offers reduced costs in deploying and maintaining the test application environment with the promise to expand and contract the test application resources easily. In addition, the test engineers can be freed from network administration to fully and completely concentrate on network protocol testing.
For more information on deploying network protocol testing applications in the cloud, contact us.