http://www.linuxpromagazine.com/Online/Features/fail2ban-Intrusion-Prevention-Framework
I dare say that only a few sys admins haven’t heard of fail2ban
– maybe those starting out or those who have focused on different
areas. In my experience, it’s quite rare that really small utilities can
affect the way you run your servers to the extent that fail2ban has. It
certainly explains its popularity.
fail2ban is a feather-weight set of scripts that can easily integrate with popular firewalls and, amongst many other things, catch any failed logins for services that you’re running and then ban the IP address after a certain number of failed attempts. Admittedly that sounds like quite simple functionality, but when you get down to the innards of the software, it’s a truly powerful tool.
I had been using fail2ban on SSH login failures, probably it’s most common usage, before I became increasingly annoyed with web server logs filling up with nefarious probes attempting to compromise PHP with remote exploits (and a myriad of other HTTP attacks). It got to the point at which a large proportion of the Apache logs were failed attempts to find hidden directories or non-existent Joomla installations among the legitimate hits on the websites.
I also ran a few mail servers that allowed mail relaying via SASL password authentication, which (and there are other ways of running the authentication side) had system user accounts with PAM checking for correct passwords. I had set the SASL user accounts so that a shell login couldn’t be used to access the server, but I was still more than aware than having a piece of software so readily open to abuse by brute force was far from ideal. So, fail2ban stepped forward yet again; I could simply ban any IP that entered the wrong password three times for as long as I wanted.
From the scenarios above, I hope you will agree that fail2ban can be applied in all sorts of ways. To give you a head start in this article, I’ll offer some examples, ranging from those straight out of the documentation to those that were hard won. (Those of you who speak regular expressions, or regex, as your second language would have found them easy, I’m sure, but I prefer a cogent language that doesn’t involve an aching head coupled with eye strain!)
It Must Be Magic
Rather than the smoke and mirrors that some pieces of software employ to keep their workings secret, fail2ban is transparent in the work it does behind the scenes.
In simple terms fail2ban keeps a close eye on your logfiles, and when a specific pattern is matched within those logs, it triggers an action that you’ve predefined within its “action” config files. The distinction between its filters and actions in the directory structure is clear once you’ve installed it.
Script Kiddies and Port 22
As I mentioned, the most popular usage of fail2ban is stopping probes that try every username ever conceived by mankind on your poor SSH server on port 22. Not only is it a logging irritation, but it is highly insecure, allowing people limitless tries at guessing your username and password combinations. Now is not the time to detail how to secure your SSH server, but for goodness sake, move it to a port other than port 22, and unless you’ve got a really, really good reason, then permit access by IP address with TCP Wrappers. Either way, you should consider banning failed logins for a short time in case a poor configuration mistake gives miscreants access inadvertently.
By my reckoning, for user-accessible services on a server, any user who gets a password wrong more than three times is going to need a new password (from a Support Department most likely), so I’d prefer not to allow nefarious types to keep trying repeatedly until they give up. Bear in mind that some scripts attack a port several times a second, so in a minute, they could have just about exhausted all popular logins beginning with the first letter of the alphabet!
Figure 1 shows an SSH logfile example of failed logins. All you
have to do (and please bear in mind my comments about eye-strain and
regex earlier) is create a regular expression to catch those failed
login attempts so you can trigger a response when they’re spotted in the
logs.
Thankfully, a few useful examples are already available post-installation, with a couple of handy tags to save you from figuring out other ways of spotting patterns – one such tag being
Fortunately, fail2ban is flexible – not just in the services it can monitor but also across different operating systems, all of which have many versions themselves. So, understanding your potential regex pain, the developers, even in the standard SSH config file, have included a number of failregex examples, which should match your system’s logfile format. The examples shown below are originally commented out in the config file, ready for you to choose one entry to copy and paste in place of the uncommented line:
SASL
I briefly touched on SASL and mail server password authentication earlier, as well as my concerns about opening up a username and password combination from /etc/shadow to the Internet as a serious hole for abuse. To allay my fears, a simple fail2ban instance is now configured to catch bad logins in a highly efficient manner.
Here is the failregex line for SASL login attempts from the config file /etc/fail2ban/filter.d/sasl.conf:
Lay of the Land
Now I’ll look a bit closer at how the config files are laid out in fail2ban’s directory structure. The above configuration files live in the filter.d/ directory. A quick ls shows a lot of out-of-the-box fail2ban filter examples. A quick warning: if they don’t fit your operating system perfectly, you might be using the search engines for a while, unless you have a diploma in regex. For many scenarios, though, the main site http://www.fail2ban.org has how-tos, as well as the bundled examples:
The main configuration file is called jail.conf, and it’s commented with lots of helpful instructions. The ignoreip setting can take CIDR IP address formats or just single IP addresses, so the Admin doesn’t get locked out accidentally; also, that’s really handy for testing. Additionally, a default setting that applies to all jails (a brief set of rules about a particular service) that don’t have the ignoreip setting explicitly configured is:
The next setting is which back-end daemon to use for polling the logfiles. In Debian’s case, I set it to gamin, which is great at checking files like mailboxes and logfiles really frequently without putting system load up.
Finally, the actions.d directory contains what actions to take and with which firewall. In the jail.conf file, other than the level of action required, to all intents and purposes, the remainder of the file is filled with a short config for each service (a jail), switched on or off with true or false. Here’s the entry for SASL:
Now that we know what to send and when to send it, we simply need to know how to ban the offending IP address. I’ll use the most popular iptables, the kernel’s Netfilter firewall. The file /etc/fail2ban/action.d/iptables-multiport.conf has these key config lines present:
Apache
Several configurations are already in place for Apache, in-hand with the mail POP3/IMAP and FTP services, but I needed a mishmash of the examples, and admittedly it took me ages to get the regex correct.
I had a very specific requirement that I knew no pages were missing on the websites on the web server (which might generate HTTP status response 404s), but I also needed to ignore explicitly some files and directories.
Any other traffic to the web server that I did not specifically allow could therefore be banned if it attempted to access files that didn’t exist or probe other parts of the server looking for PHP exploits, for example. Inside my filter.d/ directory, I created a filter called apache-misc. I’ve shortened the example a little so it’s readable, but the first line of the failregex after the [A-Z] part effectively drops any request to Apache that doesn’t include a forward slash. You might be surprised that pretty much everything legitimate should.
After the slash and inside the brackets are entries for what requests I want banned. The next line down (which is still part of the same failregex) beginning with
I’ve barely scratched the surface of this topic here, but I hope it will inspire you to look further into how fail2ban can help secure your servers. After you’ve installed it, have a look at how many IP addresses are “already banned” in the fail2ban logfile, /var/log/fail2ban.log, and see how many attempts you’ve successfully blocked in a day. With its level of reporting and the inherently efficient and controllable configuration, to my mind, it’s a knockout piece of software that should help out any sys admin greatly.
For its size, fail2ban, a utility that scans logfiles and bans suspicious IP addresses, punches well above its weight.
fail2ban is a feather-weight set of scripts that can easily integrate with popular firewalls and, amongst many other things, catch any failed logins for services that you’re running and then ban the IP address after a certain number of failed attempts. Admittedly that sounds like quite simple functionality, but when you get down to the innards of the software, it’s a truly powerful tool.
I had been using fail2ban on SSH login failures, probably it’s most common usage, before I became increasingly annoyed with web server logs filling up with nefarious probes attempting to compromise PHP with remote exploits (and a myriad of other HTTP attacks). It got to the point at which a large proportion of the Apache logs were failed attempts to find hidden directories or non-existent Joomla installations among the legitimate hits on the websites.
I also ran a few mail servers that allowed mail relaying via SASL password authentication, which (and there are other ways of running the authentication side) had system user accounts with PAM checking for correct passwords. I had set the SASL user accounts so that a shell login couldn’t be used to access the server, but I was still more than aware than having a piece of software so readily open to abuse by brute force was far from ideal. So, fail2ban stepped forward yet again; I could simply ban any IP that entered the wrong password three times for as long as I wanted.
From the scenarios above, I hope you will agree that fail2ban can be applied in all sorts of ways. To give you a head start in this article, I’ll offer some examples, ranging from those straight out of the documentation to those that were hard won. (Those of you who speak regular expressions, or regex, as your second language would have found them easy, I’m sure, but I prefer a cogent language that doesn’t involve an aching head coupled with eye strain!)
It Must Be Magic
Rather than the smoke and mirrors that some pieces of software employ to keep their workings secret, fail2ban is transparent in the work it does behind the scenes.
In simple terms fail2ban keeps a close eye on your logfiles, and when a specific pattern is matched within those logs, it triggers an action that you’ve predefined within its “action” config files. The distinction between its filters and actions in the directory structure is clear once you’ve installed it.
Script Kiddies and Port 22
As I mentioned, the most popular usage of fail2ban is stopping probes that try every username ever conceived by mankind on your poor SSH server on port 22. Not only is it a logging irritation, but it is highly insecure, allowing people limitless tries at guessing your username and password combinations. Now is not the time to detail how to secure your SSH server, but for goodness sake, move it to a port other than port 22, and unless you’ve got a really, really good reason, then permit access by IP address with TCP Wrappers. Either way, you should consider banning failed logins for a short time in case a poor configuration mistake gives miscreants access inadvertently.
By my reckoning, for user-accessible services on a server, any user who gets a password wrong more than three times is going to need a new password (from a Support Department most likely), so I’d prefer not to allow nefarious types to keep trying repeatedly until they give up. Bear in mind that some scripts attack a port several times a second, so in a minute, they could have just about exhausted all popular logins beginning with the first letter of the alphabet!
Figure 1: An example from the fail2ban site of how a standard SSH
logfile might look and what to look for when it reports failed login
attempts.
Thankfully, a few useful examples are already available post-installation, with a couple of handy tags to save you from figuring out other ways of spotting patterns – one such tag being
failregex = ^%(__prefix_line)sFailed (?:password|publickey) for .* fromBy comparing that regex with the logfile in Figure 1, I hope you get the gist of how it pieces together. As I said, don’t be too perturbed by the regex complexity; thankfully, several solid examples are included at installation time.(?: port \d*)?(?: ssh\d*)?$ ignoreregex =
Fortunately, fail2ban is flexible – not just in the services it can monitor but also across different operating systems, all of which have many versions themselves. So, understanding your potential regex pain, the developers, even in the standard SSH config file, have included a number of failregex examples, which should match your system’s logfile format. The examples shown below are originally commented out in the config file, ready for you to choose one entry to copy and paste in place of the uncommented line:
failregex = ^%(__prefix_line)s(?:error: PAM: )?Authentication failure for .* fromI still need to explain the ignoreregex line in the first example. I hope it’s clear, though, what the setting in that line, again written in hieroglyphics (sorry regex), would produce: an IP address or hostname match that should NOT trigger an action if it’s spotted within that failregex pattern in the logfiles. I’ll give you some ignoreregex examples a little later.\s*$ ^%(__prefix_line)s(?:error: PAM: )?User not known to the underlying authentication module for .* from \s*$ ^%(__prefix_line)sFailed (?:password|publickey) for .* from (?: port \d*)?(?: ssh\d*)?$ ^%(__prefix_line)sROOT LOGIN REFUSED.* FROM \s*$ ^%(__prefix_line)s[iI](?:llegal|nvalid) user .* from \s*$ ^%(__prefix_line)sUser .+ from not allowed because not listed in AllowUsers$ ^%(__prefix_line)sauthentication failure; logname=\S* uid=\S* euid=\S* tty=\S* ruser=\S* rhost= (?:\s+user=.*)?\s*$ ^%(__prefix_line)srefused connect from \S+ \( \)\s*$ ^%(__prefix_line)sAddress .* POSSIBLE BREAK-IN ATTEMPT!*\s*$ ^%(__prefix_line)sUser .+ from not allowed because none of user’s groups are listed in AllowGroups\s*$
SASL
I briefly touched on SASL and mail server password authentication earlier, as well as my concerns about opening up a username and password combination from /etc/shadow to the Internet as a serious hole for abuse. To allay my fears, a simple fail2ban instance is now configured to catch bad logins in a highly efficient manner.
Here is the failregex line for SASL login attempts from the config file /etc/fail2ban/filter.d/sasl.conf:
failregex = (?i): warning: [-._\w]+\[A quick hunt through my /var/log/mail.log files reveals:\]: SASL (?:LOGIN|PLAIN|(?:CRAM|DIGEST)-MD5) authentication failed: authentication failure
Dec 20 22:59:16 Ganymede postfix/smtpd[26681]: warning: a-not-so-innocent.ip-has-been-disguised.com[20.666.92.40]: SASL LOGIN authentication failed: authentication failureIf you squint long enough at the failregex and the mail server log format (Postfix, in this case), you can see that they should probably tie in with each other. After some testing, you should be confident that they do.
Lay of the Land
Now I’ll look a bit closer at how the config files are laid out in fail2ban’s directory structure. The above configuration files live in the filter.d/ directory. A quick ls shows a lot of out-of-the-box fail2ban filter examples. A quick warning: if they don’t fit your operating system perfectly, you might be using the search engines for a while, unless you have a diploma in regex. For many scenarios, though, the main site http://www.fail2ban.org has how-tos, as well as the bundled examples:
apache-404.conf apache-nohome.conf courierlogin.conf pam-generic.conf pure-ftpd.conf sieve.conf vsftpd.conf apache-auth.conf apache-noscript.conf couriersmtp.conf gssftpd.conf php-url-fopen.conf qmail.conf sshd.conf webmin-auth.conf apache-badbots.conf apache-overflows.conf cyrus-imap.conf lighttpd-fastcgi.conf postfix.conf sasl.conf sshd.conf.dpkg-dist wuftpd.conf apache-misc.conf common.conf exim.conf named-refused.conf proftpd.conf sasl.conf.dpkg-dist sshd-ddos.conf xinetd-fail.confAs you can see, a multitude of examples are ready to go, including a few Apache config files.
The main configuration file is called jail.conf, and it’s commented with lots of helpful instructions. The ignoreip setting can take CIDR IP address formats or just single IP addresses, so the Admin doesn’t get locked out accidentally; also, that’s really handy for testing. Additionally, a default setting that applies to all jails (a brief set of rules about a particular service) that don’t have the ignoreip setting explicitly configured is:
bantime = 1 maxretry = 3Here, it’s referring to how many opportunities a visitor has to trigger the filter; in this case, it’s set really tight and is probably impractical for most services for which users occasionally make mistakes.
The next setting is which back-end daemon to use for polling the logfiles. In Debian’s case, I set it to gamin, which is great at checking files like mailboxes and logfiles really frequently without putting system load up.
Finally, the actions.d directory contains what actions to take and with which firewall. In the jail.conf file, other than the level of action required, to all intents and purposes, the remainder of the file is filled with a short config for each service (a jail), switched on or off with true or false. Here’s the entry for SASL:
[sasl] enabled = true port = smtp filter = sasl logpath = /var/log/mail.logAside from inheriting the bantime and maxretry default setting from above, all you have to think about now is the action setting in jail.conf, which is relatively straightforward. Also, you can calm fail2ban’s reaction to a trigger; however, I prefer all guns blazing, with all the offending log entries and a whois lookup of the banned IP address mailed to me:
# ban the IP & send an e-mail with whois report and include relevant log lines to the destemail
action_mwl = %(banaction)s[name=%(__name__)s, port="%(port)s", protocol="%(protocol)s]
%(mta)s-whois-lines[name=%(__name__)s, dest="%(destemail)s", logpath=%(logpath)s]
Lights, Camera, ActionNow that we know what to send and when to send it, we simply need to know how to ban the offending IP address. I’ll use the most popular iptables, the kernel’s Netfilter firewall. The file /etc/fail2ban/action.d/iptables-multiport.conf has these key config lines present:
Actionban: actionban = iptables -I fail2ban-To anyone familiar with iptables, the commonplace switches -I for INSERT and -D for DELETE for the ban and unban settings should be pretty clear. These are editable, as you’d expect if you want to log further to manipulate iptables in some other way.1 -s -j DROP Actionunban: actionunban = iptables -D fail2ban- -s -j DROP
Apache
Several configurations are already in place for Apache, in-hand with the mail POP3/IMAP and FTP services, but I needed a mishmash of the examples, and admittedly it took me ages to get the regex correct.
I had a very specific requirement that I knew no pages were missing on the websites on the web server (which might generate HTTP status response 404s), but I also needed to ignore explicitly some files and directories.
Any other traffic to the web server that I did not specifically allow could therefore be banned if it attempted to access files that didn’t exist or probe other parts of the server looking for PHP exploits, for example. Inside my filter.d/ directory, I created a filter called apache-misc. I’ve shortened the example a little so it’s readable, but the first line of the failregex after the [A-Z] part effectively drops any request to Apache that doesn’t include a forward slash. You might be surprised that pretty much everything legitimate should.
After the slash and inside the brackets are entries for what requests I want banned. The next line down (which is still part of the same failregex) beginning with
[Definition failregex =The result of applying this filter to Apache is, whereas formerly I might have had a hundred attempts per IP probing the server, now I only have three entries in the logs (which all happened in the first second before gamin read the logfile and added the iptables rule). I’m delighted with the lack of unwanted logfile noise, and my web servers are safer! If the config is too strict for you, try and remove the 404 entry from the second failregex line and up the bantime and maxretry numbers to see how you get on..*"[A-Z]* /(cms|user|muieblackcat|db|cpcommerce|wp-login|joomla|awstatstotals|wp-content|wp-includes|pma|phpmyadmin|myadmin|mysql|mysqladmin|sqladmin|mypma|admin|xampp|mysqldb|pmadb|phpmyadmin1|phpmyadmin2).*" .*\" (502|500|417|416|415|414|413|412|404|405|403|401|400) ignoreregex = .*\"GET \/(press|mailto|domestic|word).*
I’ve barely scratched the surface of this topic here, but I hope it will inspire you to look further into how fail2ban can help secure your servers. After you’ve installed it, have a look at how many IP addresses are “already banned” in the fail2ban logfile, /var/log/fail2ban.log, and see how many attempts you’ve successfully blocked in a day. With its level of reporting and the inherently efficient and controllable configuration, to my mind, it’s a knockout piece of software that should help out any sys admin greatly.
No comments:
Post a Comment