You've found hat some process (don't know which) is eating up too much space too fast in one of your partitions and don't know where? Just yesterday I found a simple way to figure it out.
Suppose you want to check the partition where you have /home (which is in a separate partition from /, right?) where space is been eaten up too fast.
Run this simple command:
find /home -mount -type f -exec ls -s {} ';' > list_of_files1.txt
After it finishes running, wait for a little while (30 seconds, a minute, your call) and run the same thing outputting to a different file:
find /home -mount -type f -exec ls -s {} ';' > list_of_files2.txt
To find out what's going on, run a simple diff (a tool present in I guess every GNU/Linux system... if not unix) between both files and you should be able to see what's going on.
diff list_of_files1.txt list_of_files2.txt
You're welcome!
Wednesday, February 17, 2010
Tuesday, February 16, 2010
Iptables Limits Connections Per IP
How do I restrict the number of connections used by a single IP address to my server for port 80 and 25 using iptables?
You need to use the connlimit modules which allows you to restrict the number of parallel TCP connections to a server per client IP address (or address block).
This is useful to protect your server or vps box against flooding, spamming or content scraping.
Syntax
The syntax is as follows:
WARNING! Please note that large proxy servers may legitimately create a large number of connections to your server. You can skip those ips using ! syntax
You need to use the connlimit modules which allows you to restrict the number of parallel TCP connections to a server per client IP address (or address block).
This is useful to protect your server or vps box against flooding, spamming or content scraping.
Syntax
The syntax is as follows:
$ /sbin/iptables -A INPUT -p tcp --syn --dport $port -m connlimit --connlimit-above N -j REJECT --reject-with tcp-reset
# save the changes see iptables-save man page, the following is redhat and friends specific command
$ service iptables save
Example: Limit SSH Connections Per IP / HostOnly allow 3 ssg connections per client host:$ /sbin/iptables -A INPUT -p tcp --syn --dport 22 -m connlimit --connlimit-above 3 -j REJECT
# save the changes see iptables-save man page, the following is redhat and friends specific command
$ service iptables save
Example: Limit HTTP Connections Per IP / HostOnly allow 20 http connections per IP (MaxClients is set to 60 in httpd.conf):WARNING! Please note that large proxy servers may legitimately create a large number of connections to your server. You can skip those ips using ! syntax $ /sbin/iptables -A INPUT -p tcp --syn --dport 80 -m connlimit --connlimit-above 20 -j REJECT --reject-with tcp-reset
# save the changes see iptables-save man page, the following is redhat and friends specific command
$ service iptables saveSkip proxy server IP 1.2.3.4 from this kind of limitations:
$ /sbin/iptables -A INPUT -p tcp --syn --dport 80 -d ! 1.2.3.4 -m connlimit --connlimit-above 20 -j REJECT --reject-with tcp-reset
Example: Class C LimitationsIn this example, limit the parallel http requests to 20 per class C sized network (24 bit netmask)$ /sbin/iptables -A INPUT -p tcp --syn --dport 80 -m connlimit --connlimit-above 20 --connlimit-mask 24 -j REJECT --reject-with tcp-reset
# save the changes see iptables-save man page
$ service iptables save
Example: Limit Connections Per SecondThe following example will drop incoming connections if IP make more than 10 connection attempts to port 80 within 100 seconds (add rules to your iptables shell script)
#!/bin/bash IPT=/sbin/iptables # Max connection in seconds SECONDS=100 # Max connections per IP BLOCKCOUNT=10 # .... # .. # default action can be DROP or REJECT DACTION="DROP" $IPT -A INPUT -p tcp --dport 80 -i eth0 -m state --state NEW -m recent --set $IPT -A INPUT -p tcp --dport 80 -i eth0 -m state --state NEW -m recent --update --seconds ${SECONDS} --hitcount ${BLOCKCOUNT} -j ${DACTION} # .... # ..
How Do I Test My Firewall Working?Use the following shell script to connect to your web server hosted at 202.1.2.3:
#!/bin/bash ip="202.1.2.3" port="80" for i in {1..100} do # do nothing just connect and exit echo "exit" | nc ${ip} ${port}; done
References:- Lighttpd Traffic Shaping: Throttle Connections Per Single IP (Rate Limit)
- man page - iptables
Trafficanalysis Using Debian Lenny
By using my Network Monitoring Appliance we noticed a link in MRTG always under heavy load.
On this link a lot of different traffic aggregates, so we decided to analyze of what quantities of protocols and therefore applications the cumulative traffic consists.
As you can easily see, there is a lot of traffic on each day other than on weekend, and on holidays.
One OSS application which could do that is ntop.
Excerpt from www.ntop.org:
Also be aware that the following receipt describes the way we implemented our solution, I do not issue any guaranty that it works for you.
1. The Link
The Link has 10 MBit/s wirespeed and is a routing gateway between 2 sites located a few kilometers apart. A short calculation how much data travels over this link gave the result, that it should be in the range of ~25 GigaByte/Day.
On one side of the link are approximately 2000 systems, on the other side about 200, and about a dozend communication relations between both sides of interest for us.
Later we noticed that about 40-50 million packages travel over this link every day from 7am-5pm.
As ntop analyzes all traffic and shows communication relations, for example in the way of top-talkers and the like we assumed that ntop would use a lot of RAM to build tables about all communication relations we thought that the probe should have as much RAM as possible.
2. The Probe
We decided to use an old unused box, do a minimal install of Debian Lenny and use it as ntop probe. We only did a minimal install because we did not want to waste the precious RAM for X11 or other useless applications, useless for this use-case.
We decided to use Debian because it is easy adaptable to our needs, and it is known for it's stability. But you can build a probe as the described here with every other Linux Distribution you are familiar with, also the *BSD's may also be a good foundation.
We decided to place this probe near the router which handles this link, and to configure a mirror-port to get access to the whole traffic running on this link.
We equipped the probe with a second NIC, because we need one NIC for traffic-capturing, and wanted to be able to administer the probe remotely by SSH.
The first NIC (eth0) was configured without an IP adress, because it is only used to capture the traffic from the mirrorport of the router, and not to do any active communication.
Furthermore eth0 has to brought into promiscous-mode (to see all traffic), which is done by libpcap. The second NIC (eth1) which is used only for remote administration was configured with a static IP adress.
Fortunately the old box which was used as probe was equipped with 2GB RAM (which was enough), an AMD Athlon(tm) 64 Processor 3800+, and a local 80GB disc.
The disc was partioned this way:
/dev/sda1 is a 1GB partition for Lenny, /dev/sda2 a small swap-partition, and /dev/sda5 to be used for capturefiles.
Filesystems are ext3, but xfs may also be a solid candidate for such large files.
3. Offline Reporting
ntop is able to analyse traffic on the fly, or it's also able to read pcap files for later analysis and reporting.
We decided to first use the offline approach, to capture the traffic on the link with tcpdump from 7am-5pm by a short shellscript driven by cron, and to do the analysis in a second step.
The shellscipt looks like a Sys-V initscript:
Capturing traffic from 7am-5pm produces files in the size of ~30GB:
Reading a capturefile is done by
For a detailled explanation of ntop's commandline switches and parameters please have a look in it's manpage, and adjust them to your needs.
You can then examine the report of ntop in the webbrowser on port 3000 of the system where ntop is running.
4. Online Reporting
Another possibility to use ntop is that ntop itself captures the traffic on eth0 and does online reporting on the fly, so that you can see whats going on a few seconds later, and (with activated rrd-plugin) to get really nice, amazing (management friendly !) graphs where you can zoom in with your mouse.
5. First Conclusions
As expected we saw that approximately 70% of the whole traffic was induced by one system, the Internet Proxy.
Traffic associated with productive applications was in the range of 3-5%. To make the situation worse this productive traffic is associated with interactive applications, like Citrix, Telnet and SAP.
We decided that the next step should be to prioritize/shape the traffic with the help of (for instance) diffserv, or TOS mangling.
6. ngrep
We now decided to analyze the proxy related traffic, which Websites would be the top-visited ones. Such deeper analysis is not possible with ntop, but with ngrep it's easy to capture only the traffic which (for instance) goes to one system or comes/goes to one port, or much more. Ngrep is also capable to search for expressions in the payload, so the next approach was to use ngrep to capture only the Internet Proxy related traffic and further analyse it.
This is simply done by
Needless to say that this simple kind of capturing could also be done with tcpdump.
When done from 7am-5pm this could produce a file of ~14GB:
Now this file must be analyzed. The idea is
* read the file with tcpdump, dump it to stdout,
* grep for "get http://",
* cut out the FQDN of the sites,
* throw away duplicated entries which are in short series (because they may belong to one click in the browser resulting in a line of consecutive get's),
* and count and sort this data according to their occurence.
This could be done in several steps using intermediate files, or be one command where several commandline tools are piped together:
This runs for a while and you get a file sorted in descending order how often the sites found in the capturefile are visited.
I'm not 100% sure wether all our assumptions made above are handled right, but the numbers in the file look feasible.
Another way to get a report about visited Websites would be to interpret the logfile of the Internet Proxy, maybe with Calamaris (in the case of Squid or a Proxy which produces Calamaris processable logs).
Without a proxy log I'm not aware how such a report could be produced.
Last but not least we strongly like to emphasize that no user related informations have been evaluated in this project.
7. URL's
Debian
Network Monitoring Appliance
MRTG
Ntop
tcpdump + libpcap
RRD
ngrep
Squid
Calamaris
AWK 1 Liners
SED 1 Liners
On this link a lot of different traffic aggregates, so we decided to analyze of what quantities of protocols and therefore applications the cumulative traffic consists.
As you can easily see, there is a lot of traffic on each day other than on weekend, and on holidays.
One OSS application which could do that is ntop.
Excerpt from www.ntop.org:
"ntop is a network traffic probe that shows the network usage, similar to what the popular top Unix command does. ntop is based on libpcap and it has been written in a portable way in order to virtually run on every Unix platform and on Win32 as well. ntop users can use a a web browser (e.g. netscape) to navigate through ntop (that acts as a web server) traffic information and get a dump of the network status. In the latter case, ntop can be seen as a simple RMON-like agent with an embedded web interface."
Preliminary and Disclaimer
Please be aware that legal rights in some countries may forbid doing things like the ones mentioned in this article, that you do not violate legal rights.Also be aware that the following receipt describes the way we implemented our solution, I do not issue any guaranty that it works for you.
1. The Link
The Link has 10 MBit/s wirespeed and is a routing gateway between 2 sites located a few kilometers apart. A short calculation how much data travels over this link gave the result, that it should be in the range of ~25 GigaByte/Day.
On one side of the link are approximately 2000 systems, on the other side about 200, and about a dozend communication relations between both sides of interest for us.
Later we noticed that about 40-50 million packages travel over this link every day from 7am-5pm.
As ntop analyzes all traffic and shows communication relations, for example in the way of top-talkers and the like we assumed that ntop would use a lot of RAM to build tables about all communication relations we thought that the probe should have as much RAM as possible.
2. The Probe
We decided to use an old unused box, do a minimal install of Debian Lenny and use it as ntop probe. We only did a minimal install because we did not want to waste the precious RAM for X11 or other useless applications, useless for this use-case.
We decided to use Debian because it is easy adaptable to our needs, and it is known for it's stability. But you can build a probe as the described here with every other Linux Distribution you are familiar with, also the *BSD's may also be a good foundation.
We decided to place this probe near the router which handles this link, and to configure a mirror-port to get access to the whole traffic running on this link.
We equipped the probe with a second NIC, because we need one NIC for traffic-capturing, and wanted to be able to administer the probe remotely by SSH.
The first NIC (eth0) was configured without an IP adress, because it is only used to capture the traffic from the mirrorport of the router, and not to do any active communication.
Furthermore eth0 has to brought into promiscous-mode (to see all traffic), which is done by libpcap. The second NIC (eth1) which is used only for remote administration was configured with a static IP adress.
Fortunately the old box which was used as probe was equipped with 2GB RAM (which was enough), an AMD Athlon(tm) 64 Processor 3800+, and a local 80GB disc.
The disc was partioned this way:
# fdisk -l
Disk /dev/sda: 80.0 GB, 80026361856 bytes
255 heads, 63 sectors/track, 9729 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk identifier: 0x37aa37aa
Device Boot Start End Blocks Id System
/dev/sda1 1 131 1052226 83 Linux
/dev/sda2 132 162 249007+ 82 Linux swap / Solaris
/dev/sda3 163 9729 76846927+ 5 Extended
/dev/sda5 163 9729 76846896 83 Linux
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk identifier: 0x37aa37aa
Device Boot Start End Blocks Id System
/dev/sda1 1 131 1052226 83 Linux
/dev/sda2 132 162 249007+ 82 Linux swap / Solaris
/dev/sda3 163 9729 76846927+ 5 Extended
/dev/sda5 163 9729 76846896 83 Linux
/dev/sda1 is a 1GB partition for Lenny, /dev/sda2 a small swap-partition, and /dev/sda5 to be used for capturefiles.
Filesystems are ext3, but xfs may also be a solid candidate for such large files.
3. Offline Reporting
ntop is able to analyse traffic on the fly, or it's also able to read pcap files for later analysis and reporting.
We decided to first use the offline approach, to capture the traffic on the link with tcpdump from 7am-5pm by a short shellscript driven by cron, and to do the analysis in a second step.
The shellscipt looks like a Sys-V initscript:
#!/bin/sh
PATH=/sbin:/usr/sbin:/bin:/usr/bin
do_start() {
ifconfig eth0 up;
tcpdump -i eth0 -w /media/capture/`date +%F_%R`_tcpdump.pcap &
}
do_stop() {
pkill -SIGTERM tcpdump;
ifconfig eth0 down;
}
case "$1" in
start)
do_start 2>&1
;;
stop)
do_stop
;;
*)
echo "Usage: $0 start|stop" >&2
exit 3
;;
esac
Capturing traffic from 7am-5pm produces files in the size of ~30GB:
-rw-r--r-- 1 root root 32725662515 Jan 14 17:00 2020-01-14_07:00_tcpdump.pcap
Reading a capturefile is done by
ntop -m 10.80.192.0/18,10.81.20.0/24 -f /media/capture/2010-01-13_10\:30_tcpdump.pcap -n -4 -w3000 --w3c -p /etc/ntop/protocol.list
For a detailled explanation of ntop's commandline switches and parameters please have a look in it's manpage, and adjust them to your needs.
You can then examine the report of ntop in the webbrowser on port 3000 of the system where ntop is running.
4. Online Reporting
Another possibility to use ntop is that ntop itself captures the traffic on eth0 and does online reporting on the fly, so that you can see whats going on a few seconds later, and (with activated rrd-plugin) to get really nice, amazing (management friendly !) graphs where you can zoom in with your mouse.
5. First Conclusions
As expected we saw that approximately 70% of the whole traffic was induced by one system, the Internet Proxy.
Traffic associated with productive applications was in the range of 3-5%. To make the situation worse this productive traffic is associated with interactive applications, like Citrix, Telnet and SAP.
We decided that the next step should be to prioritize/shape the traffic with the help of (for instance) diffserv, or TOS mangling.
6. ngrep
We now decided to analyze the proxy related traffic, which Websites would be the top-visited ones. Such deeper analysis is not possible with ntop, but with ngrep it's easy to capture only the traffic which (for instance) goes to one system or comes/goes to one port, or much more. Ngrep is also capable to search for expressions in the payload, so the next approach was to use ngrep to capture only the Internet Proxy related traffic and further analyse it.
This is simply done by
# ngrep -d eth0 host 10.89.1.17 -O /media/capture/snap.pcap
When done from 7am-5pm this could produce a file of ~14GB:
-rw-r--r-- 1 root root 14223354675 Jan 25 16:26 snap.pcap
Now this file must be analyzed. The idea is
* read the file with tcpdump, dump it to stdout,
* grep for "get http://",
* cut out the FQDN of the sites,
* throw away duplicated entries which are in short series (because they may belong to one click in the browser resulting in a line of consecutive get's),
* and count and sort this data according to their occurence.
This could be done in several steps using intermediate files, or be one command where several commandline tools are piped together:
# tcpdump -r snap.pcap -A | grep -i "get http://" | awk '/http/ { print $2 }' | cut -d/ -f1-3 | grep http | sed '$!N; /^\(.*\)\n\1$/!P; D' | sort | uniq -c | sort -r > urls.txt
This runs for a while and you get a file sorted in descending order how often the sites found in the capturefile are visited.
I'm not 100% sure wether all our assumptions made above are handled right, but the numbers in the file look feasible.
13418 http://www.gxxxxx.dx 10184 http://www.gxxxxx-axxxxxxxx.cxx 8281 http://www.fxxxxxxx.dx 5470 http://www.bxxx.dx 4269 http://www.sxxxxxx.dx 2550 http://www.gxxx.cxx 2047 http://www.bxxxxxxx-zxxxxxx.dx 2044 http://www.fxxxxxxx.cxx 1618 http://www.exxxxxxx.dx 1410 http://www.lx-bx.dx ....
Another way to get a report about visited Websites would be to interpret the logfile of the Internet Proxy, maybe with Calamaris (in the case of Squid or a Proxy which produces Calamaris processable logs).
Without a proxy log I'm not aware how such a report could be produced.
Last but not least we strongly like to emphasize that no user related informations have been evaluated in this project.
7. URL's
Debian
Network Monitoring Appliance
MRTG
Ntop
tcpdump + libpcap
RRD
ngrep
Squid
Calamaris
AWK 1 Liners
SED 1 Liners
Subscribe to:
Posts (Atom)
