Wednesday, January 28, 2015

How to make a file immutable on Linux

http://xmodulo.com/make-file-immutable-linux.html

Suppose you want to write-protect some important files on Linux, so that they cannot be deleted or tampered with by accident or otherwise. In other cases, you may want to prevent certain configuration files from being overwritten automatically by software. While changing their ownership or permission bits on the files by using chown or chmod is one way to deal with this situation, this is not a perfect solution as it cannot prevent any action done with root privilege. That is when chattr comes in handy.

chattr is a Linux command which allows one to set or unset attributes on a file, which are separate from the standard (read, write, execute) file permission. A related command is lsattr which shows which attributes are set on a file. While file attributes managed by chattr and lsattr are originally supported by EXT file systems (EXT2/3/4) only, this feature is now available on many other native Linux file systems such as XFS, Btrfs, ReiserFS, etc.

In this tutorial, I am going to demonstrate how to use chattr to make files immutable on Linux.
chattr and lsattr commands are a part of e2fsprogs package which comes pre-installed on all modern Linux distributions.

Basic syntax of chattr is as follows.
 
$ chattr [-RVf] [operator][attribute(s)] files...

The operator can be '+' (which adds selected attributes to attribute list), '-' (which removes selected attributes from attribute list), or '=' (which forces selected attributes only).

Some of available attributes are the following.
  • a: can be opened in append mode only.
  • A: do not update atime (file access time).
  • c: automatically compressed when written to disk.
  • C: turn off copy-on-write.
  • i: set immutable.
  • s: securely deleted with automatic zeroing.

Immutable Attribute

To make a file immutable, you can add "immutable" attribute to the file as follows. For example, to write-protect /etc/passwd file:
$ sudo chattr +i /etc/passwd

Note that you must use root privilege to set or unset "immutable" attribute on a file. Now verify that "immutable" attribute is added to the file successfully.
 
$ lsattr /etc/passwd

Once the file is set immutable, this file is impervious to change for any user. Even the root cannot modify, remove, overwrite, move or rename the file. You will need to unset the immutable attribute before you can tamper with the file again.

To unset the immutable attribute, use the following command:
 
$ sudo chattr -i /etc/passwd
 

If you want to make a whole directory (e.g., /etc) including all its content immutable at once recursively, use "-R" option:
$ sudo chattr -R +i /etc

Append Only Attribute

Another useful attribute is "append-only" attribute which forces a file to grow only. You cannot overwrite or delete a file with "append-only" attribute set. This attribute can be useful when you want to prevent a log file from being cleared by accident.

Similar to immutable attribute, you can turn a file into "append-only" mode by:
 
$ sudo chattr +a /var/log/syslog

Note that when you copy an immutable or append-only file to another file, those attributes will not be preserved on the newly created file.

Conclusion

In this tutorial, I showed how to use chattr and lsattr commands to manage additional file attributes to prevent (accidental or otherwise) file tampering. Beware that you cannot rely on chattr as a security measure as one can easily undo immutability. One possible way to address this limitation is to restrict the availability of chattr command itself, or drop kernel capability CAP_LINUX_IMMUTABLE. For more details on chattr and available attributes, refer to its man page.

How to limit network bandwidth on Linux

http://xmodulo.com/limit-network-bandwidth-linux.html

If you often run multiple networking applications on your Linux desktop, or share bandwidth among multiple computers at home, you will want to have a better control over bandwidth usage. Otherwise, when you are downloading a big file with a downloader, your interactive SSH session may become sluggish to the point where it's unusable. Or when you sync a big folder over Dropbox, your roommate may complain that video streaming at her computer gets choppy.

In this tutorial, I am going to describe two different ways to rate limit network traffic on Linux.

Rate Limit an Application on Linux

One way to rate limit network traffic is via a command-line tool called trickle. The trickle command allows you to shape the traffic of any particular program by "pre-loading" a rate-limited socket library at run-time. A nice thing about trickle is that it runs purely in user-space, meaning you don't need root privilege to restrict the bandwidth usage of a program. To be compatible with trickle, the program must use socket interface with no statically linked library. trickle can be handy when you want to rate limit a program which does not have a built-in bandwidth control functionality.

To install trickle on Ubuntu, Debian and their derivatives:
 
$ sudo apt-get install trickle

To install trickle on Fedora or CentOS/RHEL (with EPEL repository):
 
$ sudo yum install trickle

Basic usage of trickle is as follows. Simply put, you prepend trickle (with rate) in front of the command you are trying to run.
 
$ trickle -d -u

This will limit the download and upload rate of to specified values (in KBytes/s).

For example, set the maximum upload bandwidth of your scp session to 100 KB/s:
 
$ trickle -u 100 scp backup.tgz alice@remote_host.com:

If you want, you can set the maximum download speed (e.g., 300 KB/s) of your Firefox browser by creating a custom launcher with the following command.
 
trickle -d 300 firefox %u

Finally, trickle can run in a daemon mode, where it can restrict the "aggregate" bandwidth usage of all running programs launched via trickle. To launch trickle as a daemon (i.e., trickled):
$ sudo trickled -d 1000

Once the trickled daemon is running in the background, you can launch other programs via trickle. If you launch one program with trickle, its maximum download rate is 1000 KB/s. If you launch another program with trickle, each of them will be rate limited to 500 KB/s, etc.

Rate Limit a Network Interface on Linux

Another way to control your bandwidth resource is to enforce bandwidth limit on a per-interface basis. This is useful when you are sharing your upstream Internet connection with someone else. Like anything else, Linux has a tool for you. wondershaper exactly does that: rate-limit a network interface.

wondershaper is in fact a shell script which uses tc to define traffic shaping and QoS for a specific network interface. Outgoing traffic is shaped by being placed in queues with different priorities, while incoming traffic is rate-limited by packet dropping.

In fact, the stated goal of wondershaper is much more than just adding bandwidth cap to an interface. wondershaper tries to maintain low latency for interactive sessions such as SSH while bulk download or upload is going on. Also, it makes sure that bulk upload (e.g., Dropbox sync) does not suffocate download, and vice versa.

To install wondershaper on Ubuntu, Debian and their derivatives:
 
$ sudo apt-get install wondershaper

To install wondershaper on Fedora or CentOS/RHEL (with EPEL repository):
 
$ sudo yum install wondershaper

Basic usage of wondershaper is as follows.
 
$ sudo wondershaper

For example, to set the maximum download/upload bandwidth for eth0 to 1000Kbit/s and 500Kbit/s, respectively:
 
$ sudo wondershaper eth0 1000 500

You can remove the rate limit by running:
 
$ sudo wondershaper clear eth0

If you are interested in how wondershaper works, you can read its shell script (/sbin/wondershaper).

Conclusion

In this tutorial, I introduced two different ways to control your bandwidth usages on Linux desktop, on per-application or per-interface basis. Both tools are extremely user-friendly, offering you a quick and easy way to shape otherwise unconstrained traffic. For those of you who want to know more about rate control on Linux, refer to the Linux bible.

Clamping down on users with rbash

http://www.itworld.com/article/2875972/clamping-down-on-users-with-rbash.html

restricted sign


One way to restrict what users can do on your Linux systems is by using rbash -- the restricted Bourne Again shell -- but only if you take some additional steps to ensure that your users can't break out of their cells.

What is rbash?

Rbash is an alternative to chrooted accounts -- though it works much better as an add-on. It works by disallowing a number of shell features. For example, when assigned rbash as their shell, your users cannot change their PATH environment variable. They cannot change directories with the cd command. They cannot use full paths to run commands. They cannot use redirection. If you meander down to the 90th page or so of the bash man page, you will likely see a list of those things that a user cannot do if they are constrained to using rbash. The list will look something like this:
  • changing directories with cd
  • setting or unsetting the values of SHELL, PATH, ENV, or BASH_ENV
  • specifying command names containing /
  • specifying a file name containing a / as an argument to the . builtin command
  • Specifying a filename containing a slash as an argument to the -p option to the hash builtin command
  • importing function definitions from the shell environment at startup
  • parsing the value of SHELLOPTS from the shell environment at startup
  • redirecting output using the >, >|, <>, >&, &>, and >> redirection operators
  • using the exec builtin command to replace the shell with another command
  • adding or deleting builtin commands with the -f and -d options to the enable builtin command
  • Using the enable builtin command to enable disabled shell builtins
  • specifying the -p option to the command builtin command
  • turning off restricted mode with set +r or set +o restricted.
That's a lot of things to not be able to do if you're an rbash user. Unfortunately (or fortunately, depending on your role in this situation), there are also a lot of ways you can escape your rbash shell if you'vr been restricted. But before we get into this, let's first look at how rbash is set up and used.

For the first thing, it's an optional feature of bash. It can be included in bash if the --enable-restricted option is used with the configure command when bash is built. To make it usable, you (assuming you're the admin now) create a symbolic link to bash and call is rbash. It's as simple as that. And if you're not sure if a system that you use or administer provides the rbash functionality, try this:
  • Create a symbolic link to bash and call it rbash -- ln -s /bin/bash rbash
  • Start rbash -- rbash
  • Try a forbidden operation such as this one -- cd /tmp
If you get a response like what you see below, rbash is available to you.
 
$ cd /tmp
rbash: cd: restricted

Now, let's try breaking out. Obviously since you're running rbash within bash, you can always just ^D your way back to your starting point, but that wouldn't prove much. Instead, try starting yet another shell.
 
$ cd /tmp
rbash: cd: restricted
$ /bin/bash
rbash: /bin/bash: restricted: cannot specify `/' in command names
$ bash
$ cd /tmp
$ pwd
/tmp
 
You can also try redirecting command output, changing your PATH variable, and asking where the date command is located (i.e., which date) and you will notice that you're not in Kansas anymore.

In the little exercise above, your restricted shell balked at your attempt to start a third shell when typing its full path, but not when you took advantage of the fact that /bin was on your search path and you typed just "bash". You might also be able to add other tools to your account that would allow you to do things that yoiur restricted environment would not.

For rbash to work well at restricting users, therefore, you also need to limit what they can do. You probably won't want /bin in their paths. That means that you probably want to set up a new bin directory containing only the commands that you want your restricted users to use. And it won't contain bash, scp, cp or any commands beyond what you want your restricted users to be able to do.

If you have a number of accounts to restrict, it's probably a good idea to set up a bin that all of them can use rather than giving each of them their own -- if only to save a little disk space.

Make sure that your restricted users' search paths are limited to your limited bin directory and, of course, make sure that bash is listed in their /etc/passwd entries.

Rbash will not stop your restricted users from listing files in other directories. If you need that kind of control, you should look into a chrooted setup in which their restricted environment appears to them to be the entire system. That kind of environment is more work than using rbash, but isn't all that difficult to configure.

How To Add Swap on FreeBSD Unix Systems

http://www.cyberciti.biz/faq/create-a-freebsd-swap-file

I need additional swap space to improve my FreeBSD Unix server/desktop performance. How do I add a swap file to FreeBSD system using command line options without creating a new partitions? How do I encrypt swap space on a FreeBSD Unix server for security purpose?

Tutorial details
DifficultyIntermediate (rss)
Root privilegesYes
RequirementsFreeBSD
Estimated completion time10m
A swap is nothing but space or file on a disk that can used as virtual memory. In FreeBSD and Unix-like operating systems, it is common to use a whole partition of a hard disk for swapping. When a FreeBSD based server runs out of memory, the kernel can move sleeping or inactive processes into swap area. A dedicated Swap partition goes a long way to avoid system freeze but if you notice you are running out of RAM or your applications are consuming too much of it then you may want to setup a swapfile. This guide helps you add a swap space on FreeBSD based Unix server.

How do I add swap on FreeBSD version 9 or older?

You will create the swap file by typing the following dd command as the root user:
dd if=/dev/zero of=/usr/swap0 bs=1M count=8192
This should create an 8GB file called swap.8G.bin in /root/. To make sure this worked you can type:
ls -alh  /root/swap.8G.bin
 
For security reason set the permissions, run:
chmod 0600 /root/swap.8G.bin
ls -alh  /root/swap.8G.bin
 
Sample outputs:
Fig.01: How to create a swap file on FreeBSD version 9.X and Earlier Commands
Fig.01: How to create a swap file on FreeBSD version 9.X and Earlier Commands

How do I activate swap space on the boot time?

To add this to your rc.conf you will type:
echo 'swapfile="/root/swap.8G.bin"' >> /etc/rc.conf
If you want to see if it is there in your rc.conf you can type:
tail /etc/rc.conf
Reboot the system:
 
reboot

A note about enabling the swap file immediately without rebooting the system

If you want to apply the swapfile immediately type the following command:
## Enable swap space ## 
mdconfig -a -t vnode -f /root/swap.8G.bin -u 0
 
## Find out configured devices i.e. swap device name ##
mdconfig -l -v
 
## Turn it on ##
swapon /dev/md0
 
Sample outputs:
Fig.02: FreeBSD find out swap device name created/attached with the mdconfig command
Fig.02: FreeBSD find out swap device name created/attached with the mdconfig command

To see details of your swap information type:
swapinfo -k
swapinfo -k | grep '/root/swap.8G.bin'
swapinfo -h
 
Sample outputs:
Device          1K-blocks     Used    Avail Capacity
/dev/ada0p3       1048540     736K     1.0G     0%
/dev/md0          8388608       0B     8.0G     0%
Total             9437148     736K     9.0G     0%

How to set up swap file on FreeBSD version 10.x or later

First, create the swap file (128M) using dd command:
dd if=/dev/zero of=/root/swap1 bs=1m count=128
Set the proper permissions on the new file for security reason:
chmod 0600 /root/swap1
Edit /etc/fstab, enter:
vi /etc/fstab
Add/append the following line:
 
## md42 will be assigned by system, use any unused device name (run 'mdconfig -lv' to get list of attached memory device names) ##
md42 none swap sw,file=/root/swap1 0 0
 
If you want to see if it is there in your /etc/fstab you can type:
tail /etc/fstab
Now, swap space will be added on system boot time. To add and activate swap space immediately, run:
 
swapon -aq
 
To see details of your swap type:
 
swapinfo -k
 
Sample session from my FreeBSD10 based server:
Fig.03: How to add  a swap file on FreeBSD version 10.x and Later
Fig.03: How to add a swap file on FreeBSD version 10.x and Later

A note about securing and encrypting swap space on a FreeBSD server

Encrypting swap space can avoid leakage of sensitive information such as passwords and other data in memory.

Procedure to encrypt swap file

Type the following command to create a swap file called /root/en.swap0
# dd if=/dev/random of=/root/en.swap0 bs=1m count=64
# mdconfig -a -t vnode -f /root/en.swap0
# geom eli init md0

Sample outputs:
Enter new passphrase:
Reenter new passphrase:
 
Metadata backup can be found in /var/backups/md0.eli and
can be restored with the following command:
 
 # geli restore /var/backups/md0.eli md0
 
Attach md0, enter:
# geom eli attach md0
Turn on encrpted swap file:
# swapon /dev/md0.eli
Verify newly created swap space:
# swapinfo -k
Sample session:
Fig.04: Encrypting swap file on a FreeBSD 10.x server
Fig.04: Encrypting swap file on a FreeBSD 10.x server

This hack is a little ugly but works. I strongly suggest that you use encrypted swap space as described here.

How can I disable devices and files for paging and swapping on FreeBSD?

Type the following command to disable /dev/md0 swap space:
# swapoff /dev/md0
# swpainfo -k

How can I display swap usage summary on FreeBSD?

Use the top command:
# top
Sample outputs (look for Swap in outputs):
 
last pid:   874;  load averages:  0.47,  0.32,  0.27                                                                                                                                                                                                    up 0+00:34:48  16:52:35
22 processes:  1 running, 21 sleeping
CPU:  0.0% user,  0.0% nice,  0.0% system,  0.0% interrupt,  100% idle
Mem: 14M Active, 13M Inact, 104M Wired, 80M Buf, 1841M Free
Swap: 1216M Total, 1216M Free
 
  PID USERNAME    THR PRI NICE   SIZE    RES STATE    TIME    WCPU COMMAND
  721 root          1  20    0 25328K  3704K select   0:00   0.00% ntpd
  755 root          1  20    0 86084K  6896K select   0:00   0.00% sshd
  765 root          1  20    0 23980K  5188K select   0:00   0.00% sendmail
  758 root          1  20    0 23492K  3452K pause    0:00   0.00% csh
....
..
 
You can also use pstat or swapinfo commands:
# pstat -s
OR
# swapinfo -k
You can also use vmstat/systat commands:
# vmstat
# systat swap

See man pages for more info:
$ man vmstat
$ man systat
$ man top
$ man swapinfo
$ man pstat

Sunday, January 25, 2015

How to build your own cloud for you and your friends on Debian Wheezy

https://www.howtoforge.com/tutorial/build-your-own-cloud-on-debian-wheezy

Get back your privacy and control over your data in just a few hours: build your own cloud for you and your friends

by Roudy Jhausse <aboutlinux@free.fr>

40'000+ searches over 8 years! That's my Google Search history. How about yours? (you can find out for yourself here) With so many data points across such a long time, Google has a very precise idea of what you've been interested in, what's been on your mind, what you are worried about, and how that all changed over the years since you first got that Google account.

Some of the most personal pieces of your identity are stored on servers around the world beyond your control

Let's say you've been a Gmail user between 2006 and 2013 like me, meaning you received 30'000+ emails and wrote about 5000 emails over that 7 year period. Some of the emails you sent or received are very personal, maybe so personal that you probably wouldn't like even some family members or close friends to go through them systematically. Maybe you also drafted a few emails that you never sent because you changed your mind at the last minute. But even if you never sent them, these emails are still stored somewhere on a server. As a result, it's fair to say that Google servers know more about your personal life than your closest friends or your family.

Statistically, it's a safe bet to consider that you've got a smartphone. You can barely use the phone without using the contacts app which stores your contacts in Google Contact on Google servers by default. So not only does Google know about your emails, but also about your offline contacts: who you like to call, who calls you, whom you text, and what you text them about. You don't have to take my word for it, you can verify for yourself by taking a look at the permissions you gave apps such as the Google Play Service to read the list of people that called you and the SMS you got. Do you also use the calendar app that comes with your phone? Unless you explicitly opted out while setting up your calendar, this means that Google knows precisely what you're up to, at every time of the day, day after day, year after year. The same applies if you chose an iPhone over an Android phone, except Apple gets to know about your correspondance, contacts and schedule instead of Google.

Do you also take great care to keep the contacts in your directory up-to-date, updating your friend's, colleagues's and and family's email addresses and phone numbers when they move to a new job or change carrier? That gives Google an extraordinarily accurate, up-to-date picture of your social network. And you love the GPS of your smartphone which you use a lot together with Google Maps.

This means Google not only knows what you do from your calendar but also where you are, where you live, where you work. And by correlating GPS location data across users, Google can also tell with whom you may socializing with right now.

Your daily habit of handing out your most personal information will impact your life in a way that no one can even forsee

To summarize, if you are an average internet user, Google has up-to-date, in-depth information about your interests, worries, passions, questions, over almost 10 years. It has a collection of some of your most personal messages (emails, SMS), an hour-by-hour detail of your daily activities and location, and a high-quality picture of your social network. Such an intimate knowledge of you likely goes beyond what your closest friends, family, or your sweetheart know of you.

It wouldn't come to mind to give this mass of deeply personal information to complete strangers, for instance by putting it all on a USB key and leaving it on a table in a random cafe with a note saying 'Personal data of Olivier Martin, use as you please'. Who knows who might find it and what they would do with it? Yet, we have no problem handing in core pieces of your identity to strangers at IT companies with a strong interest in our data (that's how they make their bread) and world-class experts in data analysis, perhaps just because it happens by default without us thinking about it when we hit that green 'Accept' button.

With so much high-quality information, over the years, Google may well get to know you better than you can ever hope to know yourself: heck, crawling through my digital past right now, I can't remember having written half of the emails I sent five years ago. I am surprised and pleased to rediscover my interest in marxism back in 2005 and my joining ATTAC (an organization which strives to limit speculation and improve social justice by taxing financial transactions) the next year.

And god knows why I was so much into dancing shoes back in 2007. These is pretty harmless information (you wouldn't have expected me to reveal something embarassing here, would you? ;-).

But by connecting the dots between high-quality data over different aspects of your life (what, when, with whom, where, ...) over such time spans, one may extrapolate predictive statements about you.

For instance, from the shopping habits of a 17-year-old girl, supermarkets can tell that she is pregnant before her dad even hears about it (true story). Who knows what will become possible with high-quality data the like Google has, which goes well beyond shopping habits? By connecting the dots, maybe one can predict how your tastes or political views will change in the coming years.

Today, companies you have never heard of claim to have 500 data points about you, including religion, sexual orientation and political views. Speaking of politics, what if you decide to go into politics in 10 years from now? Your life may change, your views too, and you may even forget, but Google won't. Will you have to worry that your opponent is in touch with someone who has access to your data at Google and can dig up something embarassing on you from those bottomless wells of personal data you gave away over the years? How long until Google or Facebook get hacked just like Sony was recently hacked and all your personal data end up in the public sphere forever?

One of the reason most of us have entrusted our personal data to these companies is that they provide their services for free. But how free is it really? The value of the average Google account varies depending on the method used to estimate it: 1000 USD/year accounts for the amount of time you invest in writing emails, while the value of your account for the advertisement industry is somewhere between 220 USD/year and 500 USD/year. So the service is not exactly free: you pay for it through advertisement and the yet unknown uses that our data may find in the future.

I've been writing about Google mostly because that's the company I've entrusted most of my digital identify to so far and hence the one I know best. But I may well have written Apple or Facebook.

These companies truly changed the world with their fantastic advances in design, engineering and services we love(d) to use, every day. But it doesn't mean we should stack up all our most personal data in their servers and entrust them with our digital lives: the potential for harm is just too large.

Claim back your privacy and that of people you care for in just 5h

It does not have to be this way. You can live in the 21st century, have a smartphone, use email and GPS on daily basis, and still retain your privacy. All you need to do is get back control over your personal data: emails, calendar, contacts, files, etc.. The Prism-Break.org website lists software that help controlling the fate of your personal data. Beyond these options, the safest and most powerful way to get back control over your personal data is to host your cloud yourself, by building your own server. But you may just not have the time and energy to research how exactly to do that and make it work smoothly.

That's where the present article fits in. In just 5 hours, we will set up a server to host your emails, contacts, calendars and files for you, your friends and your family. The server is designed to act as a hub or cloud for your personal data, so that you always retain full control over it. The data will automatically be synchronized between your PC/laptop, your phone and your tablet. Essentially, we will set up a system that replaces Gmail, Google Drive / Dropbox, Google Contacts, Google Calendar and Picasa.

Just doing this for yourself will already be a big step. But then, a significant fraction of your personal information will still leak out and end up on some servers in the silicon valley, just because so many of the people you interact with every day use Gmail and have smartphones. So it's a good idea to have some of the people you are closest to join the adventure.

We will build a system that
  1. supports an arbitrary number of domains and users. This makes it easy to share your server with family and friends, so that they get control over their personal data too and can share the cost of the server with you. The people sharing your server can use their own domain name or share yours.
  2. lets you send and receive your emails from any network upon successfully logging in onto the server. This way, you can send your emails from any of your email addresses, from any device (PC, phone, tablet), and any network (at home, at work, from a public network, ...)
  3. encrypts network traffic when sending and receiving emails so people you don't trust won't fish out your password and won't be able to read your private emails.
  4. offers state-of-the-art antispam, combining black lists of known spammers, automatic greylisting, and adaptative spam filtering. Re-training the adaptative spam filter if an email is misclassified is simply done by moving spam in or out of the Junk/Spam folder. Also, the server will contribute to community-based spam fighting efforts.
  5. requires just a few minutes of maintenance once in a while, basically to install security updates and briefly check the server logs. Adding a new email address boils down to adding one record to a database. Apart from that, you can just forget about it and live your life. I set up the system described in this article 14 months ago and the thing has just been running smoothly since then. So I completely forgot about it, until I recently smiled at the thought that casually pressing the 'Check email' button of my phone caused electrons to travel all the way to Iceland (where my server sits) and back.
To go through this article, you'll need a minimum of technical capabilities. If you know what is the difference between SMTP and IMAP, what is a DNS, and have a basic understanding of TCP/IP, you know enough to follow through. You will also need a basic working knowledge of Unix (working with files from the command line, basic system administration). And you'll need a total of 5 hours of time to set it up.

Here's an overview what we will do:
  1. Get a Virtual Private Server, a domain name, and set them up
  2. Set up postfix and dovecot to send and receive email
  3. Prevent SPAM from reaching your INBOX
  4. Make sure the emails you send get through spam filters
  5. Host calendars, contacts, files with Owncloud and set up webmail
  6. Sync your devices to the cloud

This article was inspired by and builds upon previous work

This article draws heavily from two other articles, namely Xavier Claude's and Drew Crawford's introduction to email self-hosting.

The article includes all the features of Xavier's and Draw's articles, except from three features that Drew had and which I didn't need, namely push support for email (I like to check email only when I decide to, otherwise I get distracted all the time), fulltext search in email (which I don't have a use for), and storing emails in an encrypted form (my emails and data are not critical to the point that I have to encrypt them locally on the server). If you need any of these features, feel free to just add them by following to the respective section of Drew's article, which is compatible with the present one.

Compared to Xavier's and Drew's work, the present article improves on several aspects:
  • it fixes bugs and typos based on my experience with Drew's article and the numerous comments on his original article. I also went through the present article, setting up the server from scratch several times to replicate it and make sure it would work right out of the box.
  • low maintenance: compared to Xavier's work, the present article adds support for multiple email domains on the server. It does so by requiring the minimum amount of server maintenance possible: basically, to add a domain or a user, just add one row to a mysql table and that's it (no need to add sieve scripts, ...).
  • I added webmail.
  • I added a section on setting up a cloud, to host not just your emails but also your files, your addressbook / contacts (emails, phone numbers, birthdays, ...), calendars and pictures for use across your devices.

Get a Virtual Private Server, a domain name, and set them up

Let's start by setting the basic infrastructure: our virtual private server and our domain name.

I've had an excellent experience with the Virtual Private Servers (VPS) of 1984.is and Linode. In this article, we will use Debian Wheezy, for which both 1984 and Linode provide ready-made images to deploy on your VPS. I like 1984 because the servers are hosted in Iceland which run exclusively on renewable energy (geothermical and hydropower) and hence does not contribute to the climate change, unlike the coal power plants on which most US-based datacenters currently run on. Also, they put emphasis on civil liberties, transparency, freedom and Free Software.

It could be a good idea to start a file to store the various passwords we will need to set on the server (user accounts, mail accounts, cloud accounts, database accounts). It's definitely a good idea to encrypt this file (maybe with GnuPG), so that it won't be too easy to attack your server even if the computer you use to set up your server gets stolen or compromised.

For registering a domain name, I've been using the services of gandi for over 10 years now, also with satisfaction. For this article, we will set up a zone with the name jhausse.net. We then add a host named cloud.jhausse.net to it, set the MX record to that host. While you're at it, set short Time To Lives (TTL) to your records like 300 seconds so that you'll be able to make changes to your zone and test the result rapidly while you're setting up the server.

Finally, set the PTR record (reverse DNS) so that the IP address of the host maps back to its name. If you don't understand the previous sentence, read this article to get the background. If you use Linode, you can set the PTR record in the control panel in the Remote Access section. With 1984, contact the tech support who will help you with it.

On the server, we will start by adding a non-privledged user, so that we don't end up working as root all the time. Also, to log in as root will require an extra layer of security.
 
adduser roudy

Then, in /etc/ssh/sshd_config, we set
 
PermitRootLogin no

and reload the ssh server
 
service ssh reload

Then, we'll need to change the hostname of the server. Edit /etc/hostname so that it has just a single line with your hostname, in our case
 
cloud

Then, edit the ssh server's public key files /etc/ssh/ssh_host_rsa_key.pub, /etc/ssh/ssh_host_dsa_key.pub, /etc/ssh/ssh_host_ecdsa_key.pub so that the end of the file reflects your hostname, or instance root@cloud. Then restart the system to make sure the hostname is fixed wherever is should be
 
reboot

We will update the system and remove services we don't need to reduce the risk of remote attacks.
 
apt-get update
apt-get dist-upgrade
service exim4 stop
apt-get remove exim4 rpcbind
apt-get autoremove
apt-get install vim

I like to use vim for editing config files remotely. For this, it helps to automatically turn on syntax highlighting. We do so by adding
 
syn on
to ~/.vimrc.

Set up postfix and dovecot to send and receive email

apt-get install postfix postfix-mysql dovecot-core dovecot-imapd dovecot-mysql mysql-server dovecot-lmtpd postgrey

In the Postfix configuration menu, we select Internet Site, and set the system mail name to jhausse.net.

We will now set up a database to store the list of domains hosted on our server, the list of users for each of these domains (together with their password), and a list of mail aliases (to forward email from a given address to another one).
 
mysqladmin -p create mailserver
mysql -p mailserver
mysql> GRANT SELECT ON mailserver.* TO 'mailuser'@'localhost' IDENTIFIED BY 'mailuserpass';
mysql> FLUSH PRIVILEGES;
mysql> CREATE TABLE `virtual_domains` (
  `id` int(11) NOT NULL auto_increment,
  `name` varchar(50) NOT NULL,
  PRIMARY KEY (`id`)
  ) ENGINE=InnoDB DEFAULT CHARSET=utf8;
mysql> CREATE TABLE `virtual_users` (
  `id` int(11) NOT NULL auto_increment,
  `domain_id` int(11) NOT NULL,
  `password` varchar(106) NOT NULL,
  `email` varchar(100) NOT NULL,
  PRIMARY KEY (`id`),
  UNIQUE KEY `email` (`email`),
  FOREIGN KEY (domain_id) REFERENCES virtual_domains(id) ON DELETE CASCADE
  ) ENGINE=InnoDB DEFAULT CHARSET=utf8;
mysql> CREATE TABLE `virtual_aliases` (
  `id` int(11) NOT NULL auto_increment,
  `domain_id` int(11) NOT NULL,
  `source` varchar(100) NOT NULL,
  `destination` varchar(100) NOT NULL,
  PRIMARY KEY (`id`),
  FOREIGN KEY (domain_id) REFERENCES virtual_domains(id) ON DELETE CASCADE
  ) ENGINE=InnoDB DEFAULT CHARSET=utf8;

We will host the jhausse.net domain. If there are other domains you'd like to host, you can also add them. We also set up a postmaster address for each domain, which forwards to roudy@jhausse.net.
 
mysql> INSERT INTO virtual_domains (`name`) VALUES ('jhausse.net');
mysql> INSERT INTO virtual_domains (`name`) VALUES ('otherdomain.net');
mysql> INSERT INTO virtual_aliases (`domain_id`, `source`, `destination`) VALUES ('1', 'postmaster', 'roudy@jhausse.net');
mysql> INSERT INTO virtual_aliases (`domain_id`, `source`, `destination`) VALUES ('2', 'postmaster', 'roudy@jhausse.net');

We now add a locally hosted email account roudy@jhausse.net. First, we generate a password hash for it:
 
doveadm pw -s SHA512-CRYPT

and then add the hash to the database
 
mysql> INSERT INTO `mailserver`.`virtual_users` (`domain_id`, `password`, `email`) VALUES ('1', '$6$YOURPASSWORDHASH', 'roudy@jhausse.net');

Now that our list of domains, aliases and users are in place, we will set up postfix (SMTP server, for outgoing mail). Replace the contents of /etc/postfix/main.cf with the following:
 
myhostname = cloud.jhausse.net
myorigin = /etc/mailname
mydestination = localhost.localdomain, localhost
mynetworks_style = host

# We disable relaying in the general case
smtpd_recipient_restrictions = permit_mynetworks, reject_unauth_destination
# Requirements on servers that contact us: we verify the client is not a
# known spammer (reject_rbl_client) and use a graylist mechanism
# (postgrey) to help reducing spam (check_policy_service)
smtpd_client_restrictions = permit_mynetworks, reject_rbl_client zen.spamhaus.org, check_policy_service inet:127.0.0.1:10023
disable_vrfy_command = yes
inet_interfaces = all
smtpd_banner = $myhostname ESMTP $mail_name (Debian/GNU)
biff = no
append_dot_mydomain = no
readme_directory = no

# TLS parameters
smtpd_tls_cert_file=/etc/ssl/certs/cloud.crt
smtpd_tls_key_file=/etc/ssl/private/cloud.key
smtpd_use_tls=yes
smtpd_tls_auth_only = yes
smtp_tls_security_level=may
smtp_tls_loglevel = 1
smtpd_tls_loglevel = 1
smtpd_tls_received_header = yes
smtpd_tls_session_cache_database = btree:${data_directory}/smtpd_scache
smtp_tls_session_cache_database = btree:${data_directory}/smtp_scache

# Delivery
alias_maps = hash:/etc/aliases
alias_database = hash:/etc/aliases
message_size_limit = 50000000
recipient_delimiter = +

# The next lines are useful to set up a backup MX for myfriendsdomain.org
# relay_domains = myfriendsdomain.org
# relay_recipient_maps =

# Virtual domains
virtual_transport = lmtp:unix:private/dovecot-lmtp
virtual_mailbox_domains = mysql:/etc/postfix/mysql-virtual-mailbox-domains.cf
virtual_mailbox_maps = mysql:/etc/postfix/mysql-virtual-mailbox-maps.cf
virtual_alias_maps = mysql:/etc/postfix/mysql-virtual-alias-maps.cf
local_recipient_maps = $virtual_mailbox_maps

Now we need to teach postfix to figure out which domains we would like him to accept emails for using the database we just set up. Create a new file /etc/postfix/mysql-virtual-mailbox-domains.cf and add the following:
 
user = mailuser
password = mailuserpass
hosts = 127.0.0.1
dbname = mailserver
query = SELECT 1 FROM virtual_domains WHERE name='%s'

We teach postfix to find out whether a given email account exists by creating /etc/postfix/mysql-virtual-mailbox-maps.cf with the following content
 
user = mailuser
password = mailuserpass
hosts = 127.0.0.1
dbname = mailserver
query = SELECT 1 FROM virtual_users WHERE email='%s'

Finally, postfix will use /etc/postfix/mysql-virtual-alias-maps.cf to look up mail aliases
 
user = mailuser
password = mailuserpass
hosts = 127.0.0.1
dbname = mailserver
query = SELECT virtual_aliases.destination as destination FROM virtual_aliases, virtual_domains WHERE virtual_aliases.source='%u' AND virtual_aliases.domain_id = virtual_domains.id AND virtual_domains.name='%d'

With all this in place, it is now time to test if postfix can query our database properly. We can do this using postmap:
 
postmap -q jhausse.net mysql:/etc/postfix/mysql-virtual-mailbox-domains.cf
postmap -q roudy@jhausse.net mysql:/etc/postfix/mysql-virtual-mailbox-maps.cf
postmap -q postmaster@jhausse.net mysql:/etc/postfix/mysql-virtual-alias-maps.cf
postmap -q bob@jhausse.net mysql:/etc/postfix/mysql-virtual-alias-maps.cf

If you set up everything properly, the first two queries should return 1, the third query should return roudy@jhausse.net and the last one should return nothing at all.

Now, let's set up dovecot (the IMAP server, to fetch incoming mail on the server from our devices).

Edit /etc/dovecot/dovecot.conf to set the following parameters:
 
# Enable installed protocol
# !include_try /usr/share/dovecot/protocols.d/*.protocol 
protocols = imap lmtp

which will only enable imap (to let us fetch emails) and lmtp (which postfix will use to pass incoming emails to dovecot). Edit /etc/dovecot/conf.d/10-mail.conf to set the following parameters:
 
mail_location = maildir:/var/mail/%d/%n
[...]
mail_privileged_group = mail
[...]
first_valid_uid = 0

which will store emails in /var/mail/domainname/username. Note that these settings are spread at different locations in the file, and are sometimes already there for us to set: we just need to comment them out. The other settings which are already in the file, you can leave as is. We will have to do the same to update settings in many more files in the remaining of this article. In /etc/dovecot/conf.d/10-auth.conf, set the parameters:
 
disable_plaintext_auth = yes
auth_mechanisms = plain
#!include auth-system.conf.ext
!include auth-sql.conf.ext

In /etc/dovecot/conf.d/auth-sql.conf.ext, set the following parameters:
 
passdb {
  driver = sql
  args = /etc/dovecot/dovecot-sql.conf.ext
}
userdb {
  driver = static
  args = uid=mail gid=mail home=/var/mail/%d/%n
}

where we just taught dovecot that users have their emails in /var/mail/domainname/username and to look up passwords from the database we just created. Now we still need to teach dovecot how exactly to use the database. To do so, put the following into /etc/dovecot/dovecot-sql.conf.ext:
 
driver = mysql
connect = host=localhost dbname=mailserver user=mailuser password=mailuserpass
default_pass_scheme = SHA512-CRYPT
password_query = SELECT email as user, password FROM virtual_users WHERE email='%u';

We now fix permissions on config files
 
chown -R mail:dovecot /etc/dovecot
chmod -R o-rwx /etc/dovecot

Almost there! We just need to edit a couple files more. In /etc/dovecot/conf.d/10-master.conf, set the following parameters:
 
service imap-login {
  inet_listener imap {
    #port = 143
    port = 0
  }
  inet_listener imaps {
    port = 993
    ssl = yes
  }
}

service pop3-login {
 inet_listener pop3 {
    #port = 110
    port = 0
  }
  inet_listener pop3s {
    #port = 995
    #ssl = yes
    port = 0
  }
}

service lmtp {
  unix_listener /var/spool/postfix/private/dovecot-lmtp {
    mode = 0666
    group = postfix
    user = postfix
  }
  user = mail
}

service auth {
  unix_listener auth-userdb {
    mode = 0600
    user = mail
    #group = 
  }

  # Postfix smtp-auth
  unix_listener /var/spool/postfix/private/auth {
    mode = 0666
    user = postfix
    group = postfix
  }

  # Auth process is run as this user.
  #user = $default_internal_user
  user = dovecot
}

service auth-worker {
  user = mail
}

Note that we set ports for all services but imaps to 0, which effectively disables them. Then, in /etc/dovecot/conf.d/15-lda.conf, specify an email address for the postmaster:
 
postmaster_address = postmaster@jhausse.net

Last but not least, we need to generate a pair of public and private key for the server, which we will use both in dovecot and postfix:
 
openssl req -new -newkey rsa:4096 -x509 -days 365 -nodes -out "/etc/ssl/certs/cloud.crt" -keyout "/etc/ssl/private/cloud.key"

Make sure that you specify your the Fully Qualified Domain Name (FQDN) of the server, in our case:
 
Common Name (e.g. server FQDN or YOUR name) []:cloud.jhausse.net

If you don't, our clients may complain that the server name in the SSL certificate does not match the name of the server they are connecting to. We tell dovecot to use these keys by setting the following parameters in /etc/dovecot/conf.d/10-ssl.conf:
 
ssl = required
ssl_cert = 

That's it! Now on to testing the postfix and dovecot servers!
 
service dovecot restart
service postfix restart

From the server itself, try to send an email to a local user:
 
telnet localhost 25
EHLO cloud.jhausse.net
MAIL FROM:youremail@domain.com
rcpt to:roudy@jhausse.net
data
Subject: Hallo!

This is a test, to check if cloud.jhausse.net is ready to be an MX!

Cheers, Roudy
.
QUIT

The server should accept our email with a message like
 
250 2.0.0 Ok: queued as 58D54101DB

Check the logs in /var/log/mail.log if everything went fine. There should be line saying something like
 
Nov 14 07:57:06 cloud dovecot: lmtp(4375, roudy@jhausse.net): ... saved mail to INBOX

So far so good? Good. Now, let's try the same from a different machine, like the computer we are using to set up the server. We'll talk to the server using encryption (TLS) this time:
 
openssl s_client -connect cloud.jhausse.net:25 -starttls smtp
EHLO cloud.jhausse.net
MAIL FROM:roudy@jhausse.net
rcpt to:bob@gmail.com

to which the server should respond
 
554 5.7.1 : Relay access denied

That's good: had the server accepted the mail, it would have meant that we set up postfix as an open relay for all the spammers of the world and beyhond to use. Instead of the 'Relay access denied' message, you may instead get the message
 
554 5.7.1 Service unavailable; Client host [87.68.61.119] blocked using zen.spamhaus.org; http://www.spamhaus.org/query/bl?ip=87.68.61.119

This means that you are trying to contact the server from an IP address that is considered as a spammer's address. I got this message while trying to connect to the server through my regular Internet Service Provider (ISP). To fix this issue, you can try to connect from another host, maybe another server you have access to through SSH. Alternatively, you can reconfigure Postfix's main.cf not to use Spamhaus's RBL, reload postfix, and verify that the above test works. In both cases, it's important that you find a solution that works for you because we'll test other things in a minute. If you chose to reconfigure Postfix not to use RBLs, don't forget to put the RBLs back in and to reload postfix after finishing the article to avoid getting more spam than necessary.

Now let's try to send a valid email by SMTP on port 25, which regular mail servers use to talk to each other:
 
openssl s_client -connect cloud.jhausse.net:25 -starttls smtp
EHLO cloud.jhausse.net
MAIL FROM:youremail@domain.com
rcpt to:roudy@jhausse.net

to which the server should respond
 
Client host rejected: Greylisted, see http://postgrey.schweikert.ch/help/jhausse.net.html

which shows that postgrey is working as it should. What postgrey does it to reject emails with a temporary error if the sender has never been seen before. The technical rules of email require email servers to try to deliver the email again. After five minutes, postgrey will accept the email. Legit email servers around the world will try repeatidly to redeliver the email to us, but most spammers won't. So, wait for 5 minutes, try to send the email again using the command above, and verify that postfix now accepts the email.

Afterwards, we'll check that we can fetch the two emails that we just sent ourselves by talking IMAP to dovecot:
 
openssl s_client -crlf -connect cloud.jhausse.net:993
1 login roudy@jhausse.net "mypassword"
2 LIST "" "*"
3 SELECT INBOX
4 UID fetch 1:1 (UID RFC822.SIZE FLAGS BODY.PEEK[])
5 LOGOUT

where you should replace mypassword with the password you set for this email account. If that works, we basically have a functional email server which can receive our incoming emails, and from which we get retreive these emails from our devices (PC/laptop, tablets, phones, ...). But we can't give it our emails to send unless we send them from the server itself. We'll now allow postfix to forward our emails, but only upon successful authentification, that is after it could make sure that the email comes from someone who has a valid account on the server. To do so, we'll open a special, SSL-only, SASL-authentified email submission service. Set the following parameters in /etc/postfix/master.cf:
 
submission inet n       -       -       -       -       smtpd
  -o syslog_name=postfix/submission
  -o smtpd_tls_security_level=encrypt
  -o smtpd_sasl_auth_enable=yes
  -o smtpd_client_restrictions=permit_sasl_authenticated,reject
  -o smtpd_sasl_type=dovecot
  -o smtpd_sasl_path=private/auth
  -o smtpd_sasl_security_options=noanonymous
  -o smtpd_recipient_restrictions=permit_sasl_authenticated,reject_non_fqdn_recipient,reject_unauth_destination

and reload postfix
 
service postfix reload

Now, let's try to use this service from a different machine than than the server, to verify postfix will now relay our emails and nobody else's:
 
openssl s_client -connect cloud.jhausse.net:587 -starttls smtp
EHLO cloud.jhausse.net

Notice the '250-AUTH PLAIN' capabilities advertized by server, which doesn't appear when we connect to port 25.
 
MAIL FROM:asdf@jkl.net
rcpt to:bob@gmail.com
554 5.7.1 : Relay access denied
QUIT

That's good, postfix won't relay our emails if he doesn't know us. So let's authentify ourselves first.

To do so, we first need to generate an authentification string:
 
echo -ne '\000roudy@jhausse.net\000mypassword'|base64

and let's try to send emails through the server again:
 
openssl s_client -connect cloud.jhausse.net:587 -starttls smtp
EHLO cloud.jhausse.net
AUTH PLAIN DGplYW5AMTk4NGNsb3VQLm5ldAA4bmFmNGNvNG5jOA==
MAIL FROM:asdf@jkl.net
rcpt to:bob@gmail.com

which postfix should now accept. To complete the test, let's verify that our virtual aliases work by sending an email to postmaster@jhausse.net and making sure it goes to roudy@jhausse.net:
 
telnet cloud.jhausse.net 25
EHLO cloud.jhausse.net
MAIL FROM:youremail@domain.com
rcpt to:postmaster@jhausse.net
data
Subject: Virtual alias test

Dear postmaster,
Long time no hear! I hope your MX is working smoothly and securely.
Yours sincerely, Roudy
.
QUIT

Let's check the mail made it all the way to the right inbox:
 
openssl s_client -crlf -connect cloud.jhausse.net:993
1 login roudy@jhausse.net "mypassword"
2 LIST "" "*"
3 SELECT INBOX
* 2 EXISTS
* 2 RECENT
4 LOGOUT

At this point, we have a functional email server, both for incoming and outgoing mails. We can set up our devices to use it.

PS: did you remember to try sending an email to an account hosted by the server through port 25 again, to verify that you are not longer blocked by postgrey?

Prevent SPAM from reaching your INBOX


For the sake of SPAM filtering, we already have Realtime BlackLists (RBLs) and greylisting (postgrey) in place. We'll now take our spam fighting capabilities up a notch by adding adaptative spam filtering. This means we'll add artificial intelligence to our email server, so that it can learn from experience what is spam and what is not. We will use dspam for that.
 
apt-get install dspam dovecot-antispam postfix-pcre dovecot-sieve

dovecot-antispam is a package that allows dovecot to retrain the spam filter if we find an email that is misclassified by dspam. Basically, all we need to do is to move emails in or out of the Junk/Spam folder. dovecot-antispam will then take care of calling dspam to retrain the filter. As for postfix-pcre and dovecot-sieve, we will use them respectively to pass incoming emails through the spam filter and to automatically move spam to the user's Junk/Spam folder.

In /etc/dspam/dspam.conf, set the following parameters to these values:
 
TrustedDeliveryAgent "/usr/sbin/sendmail"
UntrustedDeliveryAgent "/usr/lib/dovecot/deliver -d %u"
Tokenizer osb
IgnoreHeader X-Spam-Status
IgnoreHeader X-Spam-Scanned
IgnoreHeader X-Virus-Scanner-Result
IgnoreHeader X-Virus-Scanned
IgnoreHeader X-DKIM
IgnoreHeader DKIM-Signature
IgnoreHeader DomainKey-Signature
IgnoreHeader X-Google-Dkim-Signature
ParseToHeaders on
ChangeModeOnParse off
ChangeUserOnParse full
ServerPID               /var/run/dspam/dspam.pid
ServerDomainSocketPath  "/var/run/dspam/dspam.sock"
ClientHost      /var/run/dspam/dspam.sock

Then, in /etc/dspam/default.prefs, change the following parameters to:
 
spamAction=deliver         # { quarantine | tag | deliver } -> default:quarantine
signatureLocation=headers  # { message | headers } -> default:message
showFactors=on

Now we need to connect dspam to postfix and dovecot by adding these two lines at the end of /etc/postfix/master.cf:
 
dspam     unix  -       n       n       -       10      pipe
  flags=Ru user=dspam argv=/usr/bin/dspam --deliver=innocent,spam --user $recipient -i -f $sender -- $recipient
dovecot   unix  -       n       n       -       -       pipe
  flags=DRhu user=mail:mail argv=/usr/lib/dovecot/deliver -f ${sender} -d ${recipient}

Now we will tell postfix to filter every new email that gets submitted to the server on port 25 (normal SMTP traffic) through dspam, except if the email is submitted from the server itself (permit_mynetworks). Note that the emails we submit to postfix with SASL authentication won't be filtered through dspam either, as we set up a separate submission service for those in the previous section. Edit /etc/postfix/main.cf to change the smtpd_client_restrictions to the following:
 
smtpd_client_restrictions = permit_mynetworks, reject_rbl_client zen.spamhaus.org, check_policy_service inet:127.0.0.1:10023, check_client_access pcre:/etc/postfix/dspam_filter_access

At the end of the file, also also add:
 
# For DSPAM, only scan one mail at a time
dspam_destination_recipient_limit = 1

We now need to specify the filter we defined. Basically, we will tell postfix to send all emails (/./) to dspam through a unix socket. Create a new file /etc/postfix/dspam_filter_access and put the following line into it:
 
/./   FILTER dspam:unix:/run/dspam/dspam.sock

That's it for the postfix part. Now let's set up dovecot for spam filtering. In /etc/dovecot/conf.d/20-imap.conf, edit the imap mail_plugins plugins parameter such that:
 
mail_plugins = $mail_plugins antispam

and add a section for lmtp:
 
protocol lmtp {
# Space separated list of plugins to load (default is global mail_plugins).
  mail_plugins = $mail_plugins sieve
}

We now configure the dovecot-antispam plugin. Edit /etc/dovecot/conf.d/90-plugin.conf to add the following content to the plugin section:
 
plugin {
  ...
  # Antispam (DSPAM)
  antispam_backend = dspam
  antispam_allow_append_to_spam = YES
  antispam_spam = Junk;Spam
  antispam_trash = Trash;trash
  antispam_signature = X-DSPAM-Signature
  antispam_signature_missing = error
  antispam_dspam_binary = /usr/bin/dspam
  antispam_dspam_args = --user;%u;--deliver=;--source=error
  antispam_dspam_spam = --class=spam
  antispam_dspam_notspam = --class=innocent
  antispam_dspam_result_header = X-DSPAM-Result
}

and in /etc/dovecot/conf.d/90-sieve.conf, specify a default sieve script which will apply to all users of the server:
 
sieve_default = /etc/dovecot/default.sieve

What is sieve and why do we need a default script for all users? Sieve lets us automatize tasks on the IMAP server. In our case, we won't all emails identified as spam to be put in the Junk folder instead of in the Inbox. We would like this to be the default behavior for all users on the server; that's why we just set this script as default script. Let's create this script now, by creating a new file /etc/dovecot/default.sieve with the following content:
 
require ["regex", "fileinto", "imap4flags"];
# Catch mail tagged as Spam, except Spam retrained and delivered to the mailbox
if allof (header :regex "X-DSPAM-Result" "^(Spam|Virus|Bl[ao]cklisted)$",
          not header :contains "X-DSPAM-Reclassified" "Innocent") {
  # Mark as read
  # setflag "\\Seen";
  # Move into the Junk folder
  fileinto "Junk";
  # Stop processing here
  stop;
}

Now we need to compile this script so that dovecot can run it. We also need to give it appropriate permissions.
 
cd /etc/dovecot
sievec .
chown mail.dovecot default.siev*
chmod 0640 default.sieve
chmod 0750 default.svbin

Finally, we need to fix permissions on two postfix config files that dspam needs to read from:
 
chmod 0644 /etc/postfix/dynamicmaps.cf /etc/postfix/main.cf

That's it! Let's restart dovecot and postfix
 
service dovecot restart
service postfix restart

and test the antispam, by contacting the server from a remote host (e.g. the computer we are using to set the server):
 
openssl s_client -connect cloud.jhausse.net:25 -starttls smtp
EHLO cloud.jhausse.net
MAIL FROM:youremail@domain.com
rcpt to:roudy@jhausse.net
DATA
Subject: DSPAM test

Hi Roudy, how'd you like to eat some ham tonight? Yours, J
.
QUIT

Let's check if the mail arrived:
 
openssl s_client -crlf -connect cloud.jhausse.net:993
1 login roudy@jhausse.net "mypassword"
2 LIST "" "*"
3 SELECT INBOX
4 UID fetch 3:3 (UID RFC822.SIZE FLAGS BODY.PEEK[])

Which should return something the email with a collection of flag set by SPAM which look like this:

X-DSPAM-Result: Innocent
X-DSPAM-Processed: Sun Oct  5 16:25:48 2014
X-DSPAM-Confidence: 1.0000
X-DSPAM-Probability: 0.0023
X-DSPAM-Signature: 5431710c178911166011737
X-DSPAM-Factors: 27,
 Received*Postfix+with, 0.40000,
 Received*with+#+id, 0.40000,
 like+#+#+#+ham, 0.40000,
 some+#+tonight, 0.40000,
 Received*certificate+requested, 0.40000,
 Received*client+certificate, 0.40000,
 Received*for+roudy, 0.40000,
 Received*Sun+#+#+#+16, 0.40000,
 Received*Sun+#+Oct, 0.40000,
 Received*roudy+#+#+#+Oct, 0.40000,
 eat+some, 0.40000,
 Received*5+#+#+16, 0.40000,
 Received*cloud.jhausse.net+#+#+#+id, 0.40000,
 Roudy+#+#+#+to, 0.40000,
 Received*Oct+#+16, 0.40000,
 to+#+#+ham, 0.40000,
 Received*No+#+#+requested, 0.40000,
 Received*jhausse.net+#+#+Oct, 0.40000,
 Received*256+256, 0.40000,
 like+#+#+some, 0.40000,
 Received*ESMTPS+id, 0.40000,
 how'd+#+#+to, 0.40000,
 tonight+Yours, 0.40000,
 Received*with+cipher, 0.40000
5 LOGOUT

Good! You now have adaptive spam filtering set up for the users of your server. Of course, each user will need to train the filter in the first few weeks. To train a message as spam, just move it to a folder called "Spam" or "Junk" using any of your devices (PC, tablet, phone). Otherwise it'll be trained as ham.

Make sure the emails you send get through spam filters


Our goal in this section will be to make our mail server appear as clean as possible to the world and to make it harder for spammers to send emails in our name. As a side-effect, this will help us get our emails through the spam filters of other mail servers.

Sender Policy Framework

Sender Policy Framework (SPF) is a record that your add to your zone which declares which mail servers on the whole internet can send emails for your domain name. Setting it up is very easy, use the SPF wizard at microsoft.com to generate your SPF record, and then add it to your zone as a TXT record. It will look like this:
 
jhausse.net. 300 IN TXT v=spf1 mx mx:cloud.jhausse.net -all

Reverse PTR

We discussed this point earlier in this article, it's a good idea that you set up the reverse DNS for your server correctly, so that doing a reverse lookup on the IP address of your server returns the actual name of your server.

OpenDKIM

When we activate OpenDKIM, postfix will sign every outgoing email using a cryptographic key. We will then deposit that key in our zone, on the DNS. That way, every mail server in the world will be able to verify if the email actually came from us, or if it was forged by a spammer. Let's install opendkim:
 
apt-get install opendkim opendkim-tools

And set it up by editing /etc/opendkim.conf so that it looks like this:
 
##
## opendkim.conf -- configuration file for OpenDKIM filter
##
Canonicalization        relaxed/relaxed
ExternalIgnoreList      refile:/etc/opendkim/TrustedHosts
InternalHosts           refile:/etc/opendkim/TrustedHosts
KeyTable                refile:/etc/opendkim/KeyTable
LogWhy                  Yes
MinimumKeyBits          1024
Mode                    sv
PidFile                 /var/run/opendkim/opendkim.pid
SigningTable            refile:/etc/opendkim/SigningTable
Socket                  inet:8891@localhost
Syslog                  Yes
SyslogSuccess           Yes
TemporaryDirectory      /var/tmp
UMask                   022
UserID                  opendkim:opendkim

We'll need a couple of additional files which we will store in /etc/opendkim:
 
mkdir -pv /etc/opendkim/
cd /etc/opendkim/

Let's create a new file /etc/opendkim/TrustedHosts with the following content
 
127.0.0.1

and a new file called /etc/opendkim/KeyTable with the following content
 
cloudkey jhausse.net:mail:/etc/opendkim/mail.private

This tells OpenDKIM that we want to use an encryption key named 'cloudkey' whose contents can be found in /etc/opendkim/mail.private. We will create another file named /etc/opendkim/SigningTable and add the following line:
 
*@jhausse.net cloudkey

which tells OpenDKIM that every emails of the jhausse.net domain should be signed using the key 'cloudkey'. If we have other domains which we want to sign, we can add them here too.

The next step is to generate that key and fix permissions on OpenDKIM's config files.
 
opendkim-genkey -r -s mail [-t]
chown -Rv opendkim:opendkim /etc/opendkim
chmod 0600 /etc/opendkim/*
chmod 0700 /etc/opendkim

At first, it's a good idea to use the -t which will signal to other mail servers that you are just in testing mode, and that they shouldn't discard emails based on your OpenDKIM signature (yet). You can get your OpenDKIM key from the mail.txt file:
 
cat mail.txt

and then add it to your zone file as TXT record, which should look like this
 
mail._domainkey.cloud1984.net. 300 IN TXT v=DKIM1; k=rsa; p=MIGfMA0GCSqG...

Finally, we need to tell postfix to sign outgoing emails. At the end of /etc/postfix/main.cf, add:
 
# Now for OpenDKIM: we'll sign all outgoing emails
smtpd_milters = inet:127.0.0.1:8891
non_smtpd_milters = $smtpd_milters
milter_default_action = accept

And reload the corresponding services
 
service postfix reload
service opendkim restart

Now let's test if our OpenDKIM public key can be found and matches the private key:
 
opendkim-testkey -d jhausse.net -s mail -k mail.private -vvv

which should return
 
opendkim-testkey: key OK

For this, you may need to wait a bit until the name server has reloaded the zone (on Linode, this happens every 15min). You can use dig to check if the zone was reloaded yet.

If this works, let's test if other servers can validate our OpenDKIM signatures and SPF record. To do this, we can use Brandon Checkett's email test. To send an email to a test address given to us on Brandon's webpage, we can run the following command on the server
 
mail -s CloudCheck ihAdmTBmUH@www.brandonchecketts.com

On Brandon's webpage, you should then see result = pass in the 'DKIM Signature' section, and Result: pass in the 'SPF Information' section. If our emails pass this test, just regenerate an OpenDKIM key without the -t switch, upload the new key to the zone file, and retest to still if it still passes the tests. If so, congrats! You just successfully set up OpenDKIM and SPF on your server!

Host calendars, contacts, files with Owncloud and set up a webmail with Roundcube


Now that we have a top-notch email server, let's add to it the possibility to store your contacts, calendars, and files in the cloud. These are services that the Owncloud provides out of the box. While we're at it, we'll also set up a webmail, so you can check email even if you're travelling without electronics, or in case your phone and laptop run out of battery.

Installing Owncloud is straighforward and is well described here. On Debian, it boils down to adding the owncloud repository to your apt sources, downloading owncloud's release key and adding it to your apt keyring, and then installing owncloud itself using apt-get:
 
echo 'deb http://download.opensuse.org/repositories/isv:/ownCloud:/community/Debian_7.0/ /' >> /etc/apt/sources.list.d/owncloud.list
wget http://download.opensuse.org/repositories/isv:ownCloud:community/Debian_6.0/Release.key
apt-key add - < Release.key 
apt-get update
apt-get install apache2 owncloud roundcube

When prompted for it, choose dbconfig and then say you want roundcube to use mysql. Then, provide the mysql root password and set a good password for the roundcube mysql user. Then, edit the roundcube config file /etc/roundcube/main.inc.php so that logging in on roundcube will default to using your IMAP server:
 
$rcmail_config['default_host'] = 'ssl://localhost';
$rcmail_config['default_port'] = 993;

Now we will set up the apache2 webserver with SSL so that we can talk to Owncloud and Roundcube using encryption for our passwords and data. Let's turn on Apache's ssl module:
 
a2enmod ssl

and edit /etc/apache2/ports.conf to set the following parameters:
 
NameVirtualHost *:80
Listen 80
ServerName www.jhausse.net


    # If you add NameVirtualHost *:443 here, you will also have to change
    # the VirtualHost statement in /etc/apache2/sites-available/default-ssl
    # to 
    # Server Name Indication for SSL named virtual hosts is currently not
    # supported by MSIE on Windows XP.
    NameVirtualHost *:443
    Listen 443



    Listen 443


We'll set up a default website for encrypted connections to the webserver as https://www.jhausse.net under /var/www. Edit /etc/apache2/sites-available/default-ssl:
 

        ServerAdmin webmaster@localhost

        DocumentRoot /var/www
        ServerName www.jhausse.net
        [...]
        
          Deny from all
        
        [...]
        SSLCertificateFile    /etc/ssl/certs/cloud.crt
        SSLCertificateKeyFile /etc/ssl/private/cloud.key
        [...]


and let's also set up a website for unencrypted connections to http://www.jhausse.net under /var/www.

Edit /etc/apache2/sites-available/default:
 

        DocumentRoot /var/www
        ServerName www.jhausse.net
        [...]
        
          Deny from all
        


That way, we can serve pages for www.jhausse.net by putting them in /var/www. The 'Deny from all' directive prevents access to Owncloud through www.jhausse.net: we will set it up to access it through https://cloud.jhausse.net instead.

We will now set up the webmail (roundcube) so that it will be accessed through https://webmail.jhausse.net. Edit /etc/apache2/sites-available/roundcube to have the following content:
 


        ServerAdmin webmaster@localhost

        DocumentRoot /var/lib/roundcube
 # The host name under which you'd like to access the webmail
        ServerName webmail.jhausse.net
        
                Options FollowSymLinks
                AllowOverride None
        

        ErrorLog ${APACHE_LOG_DIR}/error.log

        # Possible values include: debug, info, notice, warn, error, crit,
        # alert, emerg.
        LogLevel warn

        CustomLog ${APACHE_LOG_DIR}/ssl_access.log combined

        #   SSL Engine Switch:
        #   Enable/Disable SSL for this virtual host.
        SSLEngine on

        # do not allow unsecured connections
        # SSLRequireSSL
        SSLCipherSuite HIGH:MEDIUM

        #   A self-signed (snakeoil) certificate can be created by installing
        #   the ssl-cert package. See
        #   /usr/share/doc/apache2.2-common/README.Debian.gz for more info.
        #   If both key and certificate are stored in the same file, only the
        #   SSLCertificateFile directive is needed.
        SSLCertificateFile    /etc/ssl/certs/cloud.crt
        SSLCertificateKeyFile /etc/ssl/private/cloud.key

        # Those aliases do not work properly with several hosts on your apache server
        # Uncomment them to use it or adapt them to your configuration
        Alias /program/js/tiny_mce/ /usr/share/tinymce/www/

        # Access to tinymce files
        
                Options Indexes MultiViews FollowSymLinks
                AllowOverride None
                Order allow,deny
                allow from all
        

        
                Options +FollowSymLinks
                # This is needed to parse /var/lib/roundcube/.htaccess. See its
                # content before setting AllowOverride to None.
                AllowOverride All
                order allow,deny
                allow from all
        

        # Protecting basic directories:
        
                Options -FollowSymLinks
                AllowOverride None
        

        
                Options -FollowSymLinks
                AllowOverride None
                Order allow,deny
                Deny from all
        

        
                Options -FollowSymLinks
                AllowOverride None
                Order allow,deny
                Deny from all
        

        
                SSLOptions +StdEnvVars
        
        
                SSLOptions +StdEnvVars
        
        #   SSL Protocol Adjustments:
        #   The safe and default but still SSL/TLS standard compliant shutdown
        #   approach is that mod_ssl sends the close notify alert but doesn't wait for
        #   the close notify alert from client. When you need a different shutdown
        #   approach you can use one of the following variables:
        #   o ssl-unclean-shutdown:
        #     This forces an unclean shutdown when the connection is closed, i.e. no
        #     SSL close notify alert is send or allowed to received.  This violates
        #     the SSL/TLS standard but is needed for some brain-dead browsers. Use
        #     this when you receive I/O errors because of the standard approach where
        #     mod_ssl sends the close notify alert.
        #   o ssl-accurate-shutdown:
        #     This forces an accurate shutdown when the connection is closed, i.e. a
        #     SSL close notify alert is send and mod_ssl waits for the close notify
        #     alert of the client. This is 100% SSL/TLS standard compliant, but in
        #     practice often causes hanging connections with brain-dead browsers. Use
        #     this only for browsers where you know that their SSL implementation
        #     works correctly.
        #   Notice: Most problems of broken clients are also related to the HTTP
        #   keep-alive facility, so you usually additionally want to disable
        #   keep-alive for those clients, too. Use variable "nokeepalive" for this.
        #   Similarly, one has to force some clients to use HTTP/1.0 to workaround
        #   their broken HTTP/1.1 implementation. Use variables "downgrade-1.0" and
        #   "force-response-1.0" for this.
        BrowserMatch "MSIE [2-6]" \
                nokeepalive ssl-unclean-shutdown \
                downgrade-1.0 force-response-1.0
        # MSIE 7 and newer should be able to use keepalive
        BrowserMatch "MSIE [17-9]" ssl-unclean-shutdown



and declare the server in your DNS, for instance:
 
webmail.jhausse.net. 300 IN CNAME cloud.jhausse.net.

Now let's enable these three websites
 
a2ensite default default-ssl roundcube
service apache2 restart

and the webmail, accessible under https://webmail.jhausse.net, should basically work. Log in using the full email (e.g. roudy@jhausse.net) and the password you set in mailserver DB at the beginning of this article. The first time you connect, the browser will warn you that the certificate was not signed by a certification authority. That's fine, just add an exception.

Last but not least, we will create a virtual host for owncloud by putting the following content in /etc/apache2/sites-available/owncloud:
 


        ServerAdmin webmaster@localhost

        DocumentRoot /var/www/owncloud
        ServerName cloud.jhausse.net
        
                Options FollowSymLinks
                AllowOverride None
        
        
                Options Indexes FollowSymLinks MultiViews
                AllowOverride All
                Order allow,deny
                allow from all
        

        ScriptAlias /cgi-bin/ /usr/lib/cgi-bin/
        
                AllowOverride None
                Options +ExecCGI -MultiViews +SymLinksIfOwnerMatch
                Order allow,deny
                Allow from all
        

        ErrorLog ${APACHE_LOG_DIR}/error.log

        # Possible values include: debug, info, notice, warn, error, crit,
        # alert, emerg.
        LogLevel warn

        CustomLog ${APACHE_LOG_DIR}/ssl_access.log combined

        #   SSL Engine Switch:
        #   Enable/Disable SSL for this virtual host.
        SSLEngine on

        # do not allow unsecured connections
        # SSLRequireSSL
        SSLCipherSuite HIGH:MEDIUM
        SSLCertificateFile    /etc/ssl/certs/cloud.crt
        SSLCertificateKeyFile /etc/ssl/private/cloud.key

        
                SSLOptions +StdEnvVars
        
        
                SSLOptions +StdEnvVars
        

        BrowserMatch "MSIE [2-6]" \
                nokeepalive ssl-unclean-shutdown \
                downgrade-1.0 force-response-1.0
        # MSIE 7 and newer should be able to use keepalive
        BrowserMatch "MSIE [17-9]" ssl-unclean-shutdown



and activate owncloud by running
 
a2ensite owncloud
service apache2 reload

Then go ahead an configure owncloud by connecting to https://cloud.jhausse.net/ in a web browswer.
That's it! Now you've got your own Google Drive, Calendar, Contacts, Dropbox, and Gmail! Enjoy your freshly recovered privacy! :-)

Sync your devices to the cloud


To sync your emails, you can just use your favorite email client: the standard email program on Android or iOS, k9mail, or Thunderbird on your PC. Or you can also use the webmail we set up.

How to sync your calendar and contacts with the cloud is described in the doc of owncloud. On Android, I'm using the CalDAV-Sync and CardDAV-Sync apps which act as bridges between the Android calendar and contacts apps of the phone and the owncloud server.

For files, there is an Android app called Owncloud to access your files from your phone and automatically upload pictures and videos you take to your cloud. Accessing your files on the your Mac/PC is easy and well described in the Owncloud documentation.

Last tips


During the first few weeks, it's a good idea to monitor /var/log/syslog and /var/log/mail.log on a daily basis and make sure everything everything is running smoothly. It's important to do so before you invite others (friends, family, ...) to be hosted on your server; you might loose their trust in self-hosting for good if they trust you with their data and the server suddently becomes unavailable.

To add another email user, just add a row to the virtual_users table of the mailserver DB.

To add a domain, just add a row to the virtual_domains table. Then update /etc/opendkim/SigningTable to get outgoing emails signed, upload the OpenDKIM key to the zone, and reload OpenDKIM.

Owncloud has its own user DB which can be managed by logging in in Owncloud as administrator.

Finally, it's important to think in advance of a solution in case your server becomes temporarily unavailable. For instance, where would your mails go until your server returns? One solution would be to find a friend who can act as your backup MX, while you act as his backup MX (see the relay_domains and relay_recipient_maps setting in Postfix's main.cf file). Similarly, what if your server is compromised and a malicious individual erases all your files there? For that, it's important to think of a regular backup system. Linode offers backups as an option. On 1984.is, I set up a basic but sufficient automatic backup system using on crontabs and scp.

Women of OpenStack, a guide to testing, and more

http://opensource.com/business/15/1/openstack-news-january-12

Blue clouds, OpenStack
Image by : 
opensource.com
Interested in keeping track of what's happening in the open source cloud? Opensource.com is your source for what's happening right now in OpenStack, the open source cloud infrastructure project.

OpenStack around the web

There's a lot of interesting stuff being written about OpenStack. Here's a sampling:
And just for fun...

OpenStack discussions

Here's a sample of some of the most active discussions this week on the OpenStack developers' listserv. For a more in-depth look, why not dive in yourself?

OpenStack events

Here's what is happening this week. Something missing? Email us and let us know, or add it to our events calendar.
Interested in the OpenStack project development meetings? A complete list of these meetings' regular schedules is available here.