Open Source Does Not Equal Better Quality or Greater Security
Open source advocates have long proclaimed the intrinsic quality and security of open source code. They argue that because the code is open it is inspected by many eyes and tested by many hands.
I dispute that argument.
Code will become better through more inspection, and improved testing, no matter whether that code is "open", "free", or "proprietary".
Background
I worked for several years building "secure" operating systems and networks for various three and four letter government agencies in the US and UK. Several of these efforts involved formal proofs that the code and hardware complied with mathematically expressed security models.
I've also participated on "tiger teams" to penetrate (with permission) various systems.
And I've been writing code — often code to test other code — for several decades. (I began in the era of punched cards.)
So I've seen a lot of code flaws; I've created many of them myself.
It is my strongly held belief that the claims that open source software, simply because it is open source, is more secure or of better quality than other software, are claims that are both overbroad and somewhat off target.
A Diversion - What Is Open Source
An open source (or free software) license is nothing more than a simple contract, an agreement that the law will enforce. That contract arises from an agreement (usually tacit or implied) between the provider, the licensor, and the taker, the licensee.
Most open source license agreements grant the right to make copies of the software, to use the software, to make changes, and to redistribute the code with those changes.
Most of the debate and dispute surrounding open source is focused on only one of those rights: redistribution.
However, from the point of view of software quality and security, the grant of the right to use the software is the most important. It is from that right that we gain the ability to inspect and test code.
This note argues that a full open source license is not necessary for the purpose of improving software quality and security. Rather, the argument here is that it is necessary only to allow the licensee (and any further downstream licensees) to inspect the code, to test it, and to publish the results of that testing. (We will carry this argument further, by suggesting that testing that is merely informal or ad hoc is also insufficient.)
On the Nature of Software (and Hardware) Flaws
Flaws that should be obvious are often hard to see.
And many of our programming languages (particularly C and C++) have aspects, and historical coding styles, that favor terse expression over clarity of code for inspection.
(In C/C++ the worse of these are the easily confused "=" versus "==" operators and coding styles that often do not bother to put overt '{' and '}' brackets after conditionals.)
In addition to syntax based problems, code flaws are often hidden more deeply; parameters are often not fully checked or there may be logic flaws. And in today's increasingly multi-threaded world there is often inadequate locking against race conditions. Logic bugs can be hard to find, particularly in multi-threaded and network code.
In many cases flaws appear only when the software is operated under conditions that do not typically occur on developers' systems and networks but which can, and do, happen in real life either as the result of errors, noise, or hostile users.
The Recent libssh Bug
Recently a major flaw, CVE-2018-10933, was discovered in the widely used open source libssh. This bug opens the door to logins without proper passwords or credentials. The source code was patched quickly after the bug was discovered. Considerable time and money will be required to apply that fix to many already deployed systems and products. Many devices may never be patched.
(There is confusion about the presence or absence of this flaw in any given product; much depends on choices such as whether a given program uses "libssh" (some version flawed), "libssh2" (without this flaw), or some other SSH code.)
The important point here is that this flaw has been there for all to see for several years.
This code forms the basis of one of the main security tools of the internet. This code is written by, updated by, inspected by, and used by people who are deeply skilled in matters of computer security.
This code is open source; anyone could have looked at it and tested it.
Yet the bug remained, undiscovered for several years.
Real Inspection and Real Testing
So what can be done?
Programing languages can be improved by eliminating hard-to-see syntax; by requiring more formalized definitions of variables and parameters; by better controlling and validating pointers; by imposing more formality on the allocation, use, and release of dynamically allocated memory; and by incorporating many other good ideas that have been posited over the years and deployed into newer languages. This is, however, a long-term measure, and it is something that will always face questions regarding the cost in terms of memory, speed, and portability, especially in the Internet of Things (IoT) world where costs and computational resources can be severely constrained.
Programmers can voluntarily adopt programming rules. For example by adopting a rule that all conditional statements in C/C++ (and similar languages) be wrapped in explicit code brackets. Or it can be more esoteric by adopting a rule that requires code to explicitly express (perhaps if only in a comment) the fate of memory passed by reference (a pointer). (Languages such as Rust have this built in.)
When we were designing and coding secure operating systems, our programming rules obligated us to clearly define for each parameter whether it represented an immutable value/object or something that could be changed. (This would, for instance, be done by obligating C/C++ code to more rigidly adhere to proper and full use of the "const" keyword — something that, unfortunately, tends to break many library API calls.)
Programmer toolkits can provide more (and better) syntax "linters", memory tracking, and flow analysis tools.
Code should be better tested. This means not only better test tools, but also adopting those tools into a procedural and administrative system that encourages frequent testing and obligates programmers to respond to test results. In many companies the Q/A test team is often viewed with less esteem than the programmers. Q/A staff ought to have a degree of veto power to block code releases.
Attention should be paid to "negative testing", which is testing to assure that the code properly handles incorrect inputs and conditions.
A good discussion of the various kinds of testing can be found in IWL's Network Protocol Testing Overview.
The libssh source code build procedure automatically runs the code against a suite of "torture" test cases. That was good, but it was not good enough. Those test cases were not sufficiently imaginative to encompass the kind of situation where libssh failed when its network peer intentionally inserted an unexpected step or value into the protocol exchange.
In other words, test cases are often written from the viewpoint of the programmer who wants to assure that the code does what it is supposed to do. That is necessary, but it is not enough. Rather, there should be tests in which the intention is to evaluate the code in situations that are not routine, or are hostile.
It is always useful to have ad hoc testing in which a test is made of whatever is of interest at the moment. However, it is important that there be a comprehensive testing plan that tries to provide a sieve of tests to improve the chance that flaws will be discovered.
When you test drive an automobile, you check to see how it accelerates, how it handles corners, rear window visibility for parallel parking, and that sort of thing. Your test drive might take 30 minutes. That is ad hoc testing.
However, you would most likely be very concerned if that was the same sort of testing the automobile manufacturer was conducting. In other words, you expect much more thorough, comprehensive testing from the automotive manufacturer and you expect certifications about crash worthiness, EMC testing, noise measurements, high altitude and bad weather performance, battery safety, and so on. You most certainly would be disappointed if you learned that the manufacturer had tested only on smooth roads on pleasant sunny days.
This is where IWL's Maxwell® and KMAX® products come into play. Those products are tools that are intended to provide code testers with means to introduce controlled, out-of-norm, network conditions and erroneous data exchanges. IWL's products can be used in both ad hoc tests and comprehensive test systems.
Conclusion
We should be careful to understand exactly what it is that we require of our software and hardware designs.
In terms of security and trustworthiness it can be more important that code and designs be open to testing and publication of test results than it is that the code and designs be freely available to all.
IWL's Maxwell® and KMAX® products are tools that ought to be among those used for testing network based code.