112 Chapter 3 · A Tangled Web Testing HTTP-scanning Solutions Proper testing of a perimeter security solution is not trivial. Currently, there seem to be no tests that actually compare perimeter protection solutions such as antivirus or Intrusion Prevention Systems (IPS) or both on anything but a set of features (IP-blocking, content filtering features, and so on.) rather than attack samples. Surprisingly, we found no tests that included a comparison of detection rates and levels of proactive exploit blocking.The reason is fairly obvious: proper comparative testing of perimeter solutions is a very non-trivial job. First, it is really very different from testing traditional AV solutions because the target objects are not files, but network transmissions. Second, finding false negatives is very tricky, because for gateway solutions proactive protection is very commonplace. (IPS provides proactive protection while AV features both reactive and proactive detection). Finally, finding false positives is very hard. Perhaps the easiest test to implement is performance measurement. But even that is far from trivial. First, there is the problem of selecting a representative test set. And a network pattern within any real network may vary greatly from the one used in testing. Second, throughput and latency are interrelated and thus difficult to separate in a test. One big mistake that can be made is simply to use samples from an AV test set to eval- uate perimeter products.You might think that if all the HTML samples are collected from all available AV collections, that this would make up a good representative sample set for testing the performance of an HTTP-scanning device. Not at all!