Use Profiling to Improve Snort Performance

Snort, the open source intrusion detection and prevention system (IDS/IPS), can be a fabulous tool to protect your network from attack – if it’s set up correctly. If it’s not, it can cause network traffic and performance problems. Here’s a rundown of how to use Snort’s built-in profiling tools to tune your setup and improve Snort’s performance.
Snort is generally used to monitor and analyze incoming network traffic, to detect potential probes and attacks of various sorts. Whilst the main powerhouse of Snort is the detection engine, not all attacks can be identified here, so it also has an array of preprocessors that either look at packets themselves or modify traffic before passing it to the detection engine. Obviously, this kind of analysis takes some system resources, and Snort can cause delays in your network traffic if it is not performing well. Inevitably, tuning Snort forces you to balance between the risk of intrusion and maintaining a smoothly functioning network, but by monitoring performance and tuning it carefully to your own systems and requirements, you can do your best to maximize both.
Snort provides its own performance monitoring tool, perfmon, as a preprocessor, with a long list of options. You can start using perfmon by adding a line to snort.conf:

preprocessor perfmonitor: time 300 flow events file snortfile perfstats.log

flow shows statistics about the type of traffic and protocols that Snort is seeing; events shows statistics about how many rules were evaluated and matched, and how many were evaluated and did not match. The file and snortfile options say to log raw data to the file perfstats.log in the Snort log directory; alternatively, using console rather than the file option outputs nicely formatted data to the console. console output is readily human-readable, while file output is more useful for graphing or other analysis, so you might want to use both at the same time. You’ll need to run Snort from the command line to get console output; if you want to save that to a text file for later examination, you’ll need to redirect stdout:

/usr/bin/snort -c /etc/snort/snort.conf -i [interface] 2>&1 > CONSOLE.log

If you have a low-traffic network, you may want to use the pktcnt keyword to reduce the number of packets required to pass through Snort before it checks for the time sample (the default is 10,000):

preprocessor perfmonitor: time 60 pktcnt 500

Note that after any config changes you need to restart Snort.

Looking at the perfmon Results

For human-readable results, you need to look at the console output:

The first group of results shows some of the basic metrics, including drop rate, megabits per second, and bytes per packet. The better Snort is performing, the lower the packet drop rate will be, the higher the Mbits/Sec throughput will be, and the higher the average packet size will be. If any of these indicate low performance, you should tune your Snort instance using some of the suggestions below. A packet drop rate of over 10% is definitely considered to be problematic, but really you want it to be as close to 0% as possible. (Note that some packets will always be dropped on startup.) Throughput will vary depending on your network. If you’re seeing poor performance, you may also want to consider increasing the system resources available (CPU, memory, etc).
An important statistic to look for is the pattern match percentage, which shows the percentage of bytes that Snort is sending through the pattern matcher. In theory, this number could be higher than 100%, if packets are being defragmented or reassembled, but for optimum performance, it should be somewhere around 10%. If it’s higher than you want, try these changes:

• Stream TCP reassembly, performed by the stream5_tcp preprocessor, aims to defeat attacks that involve forged fragmented packets. To improve performance, check that the protocol client option is set, which means that only sessions originating client-side are reassembled, and consider adjusting the list of ports for which sessions are reassembled by using the ports option to check only the ports you are interested in.
• Normal Snort can’t usefully inspect encrypted traffic, so making sure that the SSL and SSH preprocessors are turned on should reduce the amount of traffic Snort needs to inspect. If the flow data shows that you have significant traffic on encrypted ports, these preprocessors are definitely needed.
• Since all known exploits occur at the start of an SSH session, the default setting for the SSH preprocessor is to only inspect the first 25 packets of an encrypted session. You can consider decreasing this number to improve performance, but be warned that decreasing it too far may increase false negatives. A value of 20 is a reasonable compromise. If you’re seeing false positives for Challenge-Response Overflow or CRC32, try increasing max_client_bytes from its default of 19600 to wait for longer before alerting on the basis of unanswered bytes. Bump it up a little at a time until the rate of false positives falls.
• The SSL preprocessor can be set to stop inspection of encrypted traffic, thus increasing performance without affecting results, by turning the noinspect_encrypted option on (it’s off by default). If you trust that your own servers are not compromised, you can also improve performance by turning trustservers on, thus checking only for successful encryption client-side.
• Look at the (numerous!) HTTP Inspect options, in particular client_flow_depth and server_flow_depth. Too small a server_flow_depth may cause false negatives; but most HTTP server response rules are targeted at either the header, or the first 100-odd bytes of non-header data. The default setting for this, 300, will usually catch the header. The client_flow_depth option is used primarily to avoid Snort inspecting large cookies at the end of HTTP client request headers. Note that if either client_flow_depth or server_flow_depth is set to 0, Snort will inspect all client-side or server-side traffic, which will slow performance; if client_flow_depth is not set to zero, rules inspecting data beyond the first 1460 bytes of the first packet of a client request will not be effective (and so may as well be eliminated). With these settings, it’s a balance between checking traffic and maintaining performance.

In the next group of statistics you’ll see on the console, TCP Stream metrics, the main issue is the timeout rate. If this is high (most likely if you have heavy network traffic), try raising your timeout setting, say by 30 seconds at a time, to improve performance.
perfmon will show you fragmentation statistics, but consider turning off the frag3 preprocessor, as it does have a performance cost. Frag3 uses target-based analysis, modeling the network targets rather than just the looking for attacks within the protocols, and is a relatively new introduction. If you want to keep frag3, preallocating fragment nodes with prealloc_memcap or prealloc_frags will change the memory management model, and may help to reduce overhead.

Perfmon CSV Data

If you don’t use console, your perfmon data appears in lots of CSV numbers, which basically aren’t human-parseable, though if you really want to try, the fields are (not very well) documented in section 2.2.5 of the manual.
Note that some fields, such as drop-rate, are averaged over the lifetime of the Snort process, while others are managed over the data collection period. It may be easiest just to restart the Snort process when you start and stop collecting data, so you’re clear on what you’re looking at.
To visualise the CSV data, try using RRDtool or another graphing application. A visual representation of the data enables you to see spikes in alerts, dropped packets, and flow information about port traffic. You can choose to not process data through a particular port with the ignore_ports Snort config setting. You might want to do this if you have local known-safe traffic on a particular port, for example. Whether it’s worth doing this depends on what your traffic flow through specific ports looks like.

Snort Performance Profiling

Snort handles traffic in two ways: via preprocessors and via rules in the detection engine. Performance profiling gives you statistics on the performance of both the preprocessors and your rules, enabling you to find performance bottlenecks and fine-tune your setup.
To enable performance profiling, you must have compiled Snort with the -enable-perfprofiling option. If you installed Snort from a precompiled package, check your snort.pc file, possibly in /usr/local/lib/pkgconfig, to see if the option is set. Then edit snort.conf to add:

config profile_rules: filename snort_perflog.txt append

This setting causes Snort to print all performance statistics to the file snort_perflog.txt. (See the Snort manual for more options.) If you leave off append, the output will go to a different file each time, with a timestamp added to the root name.
The preprocessor profiling output should give you some idea of how much CPU time each preprocessor component is taking up, and thus where to focus your tuning efforts. Some of the suggestions above may help with particular preprocessors.
The performance profile on the rules identifies which your most CPU-intensive rules are. The first important thing to bear in mind is that if a rule is causing alerts, it’s probably doing an important job and should at least initially be left alone; look for other performance problems first before considering optimization of successful rules.
Otherwise, a high “check” count indicates a frequent content match, meaning the rule is being run a lot; you might want to see if you can tune it to run it less often. A high “Avg/check” value often means an expensive regular expression – that is, one which requires a lot of processor power; see if you can improve the regexp. A particularly egregious example for problem regexps is catastrophic backtracking, but in general it is worth optimizing expensive regexps if you experience performance problems.
If Snort is evaluating lots of rules and not matching many, you probably want to look at the detail of your rules. Wherever possible, include a content option in a rule, as rules without this option are always evaluated. Including content means that the rules are filtered via the fast pattern matcher, and thus that they are more likely to be applicable when they’re evaluated, improving performance.
If a rule matches often but doesn’t alert, it may well be setting flowbits. A flowbit enables you to track multiple packets in the same flow; rule 1 will set the flowbit, and rule 2 will look for both a rule match and the flowbit from rule 1. Examine any expensive rules; if you don’t care about a particular flowbit, you can disable the rules that set and check it.
Finally, bear in mind that performance profiling itself has some performance cost! You are unlikely to want to leave either perfmon or perfprofiling on all the time; use them for just long enough to get useful data.