Software development has always been one of the weak points with respect to security: a lot of attacks consist of bug exploits of traditional and wide-spread software. Assurance and functionality do not go together well. Feature-rich software tends to be complex and hard to test. Simple software is probably easier to test, but not very flexible and has limited functionality.
The more profound the analysis of security related events has to be, the more complex the rules and the engines need to be, the more vulnerable they become for design flaws and the more resources they will need. Again a trade-off has to be made between performance, functionality and accepted risk.
There are really only two ways to try to determine the correctness of a system: through testing and through verification. Verification is aimed at detecting flaws in the design using logical reasoning. Testing is done on real implementations of the design.
The problem extends beyond the adaptive security management system's design though. Slightly different implementations of standard protocols, interfaces and so on can be a nightmare for security officers and developers of safeguards4.1. A notorious example are TCP/IP implementations from various vendors. Although the Internet protocols TCP, UDP, IP, ICMP, etc. are standardized by the Internet Engineering Task Force (IETF) in the form of Request for Comments (RFCs), many software vendors have implemented the standards poorly when it comes to details that only change the behaviour of the code in rare circumstances. Little details that can have big consequences for security. An example of a vulnerability is the different behaviour of the code that has to reassemble overlapping IP (or TCP) packets in several different operating systems. The standard suggests that incoming data forward overlaps existing data, and that it should not reverse overlap existing data (if ``ATTA_K'' were received first, and then ``CH'' for the last two positions, the system should assemble the data to ``ATTACH'' and not ``ATTACK''; if the second packet consisted of ``IC'' for positions 4 and 5, the system should see ``ATTACK'' and not ``ATTICK''). Microsoft Windows NT 4.0 (for IP and TCP) and Sun Solaris 2.6 (for IP) however appear to always favour old data. (The example is taken from [PTA98, pp.21-22 and 34], who describes in detail how rewriting packets streams can fool intrusion detection systems that reassemble these packets in a different fashion to the destination machines).
Evaluation of systems and independent testing for this type of vulnerabilities could justify a user's trust in the assurance the products claim to offer. Scanners can help in detecting these anomalies as well. ``Security through obscurity'' is certainly also a way to hide vulnerabilities. Source code may reveal much about the internal working and behaviour of the software to hackers. It is probably however only a matter of time before someone by accident finds a way to exploit a hidden bug (or feature). Making source code publicly available increases the likelihood of someone with non-malevolent intentions spotting potential problems.
It is important to note the difference between a host being compromised and the security system itself being subverted. If an intruder obtains unauthorized access to a computer host, he may well be detected on his next move, be it on the computer or on a network connection. When the security management system itself is defeated, all bets are off: the ASMS can even become a versatile tool for the attacker. He can fool the security manager by feeding the system with incorrect information; he can create a false sense of security, giving himself more time to do whatever he is up to; he can even draw the attention to a different area than the target of his attacks.