Build an IPSEC VPN Without Losing Your Mind

With latency-sensitive or high bandwidth requirements, often we find OpenVPN unable to perform. IPSEC is the solution, but the barrier to using IPSEC is great.

It is potentially difficult to configure, and one's first exposure to VPN concepts is often confusing.

With the concepts explained, the task of actually making IPSEC work becomes much more tenable.

Likewise, using Openswan, the fork and improvement of FreeS/WAN, makes configuring IPSEC almost fun.

We will explain how to create a network-to-network connection so hosts at two remote sites can communicate with each other.

If you have trouble setting this up, first try host-to-host mode, using the excellent tutorial on the Openswan wiki.

Concepts
To successfully run Openswan, there are only four things you need:

1. a pre-shared key to make encryption work
2. to know the IP addresses of the two gateway servers on each end
3. to know the subnet ranges of the networks at each end
4. a drawing, if this is your first time

You can ignore requirement number one, for now, since Openswan makes this simple.

Openswan also takes care of adding routing table entries, so this list is really all you need for the basic "make one subnet reach another" configuration.

After configuring a tunnel, Openswan will inherently know which traffic is destined for the network on the other side of the tunnel, and automatically adds the route.

Of course, you can send traffic for more subnets through the IPSEC tunnel, assuming the gateway on the other end knows how to route there.

When everything else is working, simply add another route similar to the one Openswan adds for you, but for the other network.

When configuring IPSEC, you need to draw out your subnets and which interfaces are on the gateways, and label them accordingly.

Using the intuitive left/right terminology from Openswan (because all diagrams of this nature are two-dimensional, and in/out or local/remote mean different things to different people), create a diagram similar to the figure below with your own information:

The site represented on the left side, with Gateway A, is known as the "left" when we configure Openswan, and likewise for the right.

How-To

We are now ready. Install Openswan and locate its ipsec.conf file.

If a key was generated by your installer package, you can retrieve it by running

# ipsec showhostkey --left

We will use the resulting key for the configuration file. If instead you see something like "ipsec showhostkey: no default key in /etc/ipsec.secrets," you will need to generate one as root: ipsec newhostkey --output /etc/ipsec.secrets


Repeat the process on the other side of the soon-to-be VPN, and you should now have two keys.

The ipsec.conf file is really quite simple with Openswan. We are now going to combine all the information previously gathered above.


# Openswan ipsec.conf
conn GatewayA-to-GatewayB
left=4.3.2.1
leftsubnet=10.1.0.0/24
leftid=@a.example.com
leftrsasigkey=0sAQNe...
leftnexthop=%defaultroute
right=1.2.3.4
rightsubnet=10.2.0.0/24
rightid=@b.example.com
rightrsasigkey=7xoprFh...
rightnexthop=%defaultroute
auto=start

The "left" and "right" declarations are externally accessible IP addresses of the gateway routers. The "leftid" is the hostname of the machine.

Copy this file to Gateway B, and you are ready to start the tunnel on both ends:

# ipsec auto --up GatewayA-to-GatewayB

The quick test is of course to ping the external IP of Gateway B from Gateway A. You will notice a route in the routing table that directs this traffic, and traffic to the defined rightsubnet through the tunnel.

If IP forwarding is enabled, as it probably is, given that these machines are both already routers, you can now try pinging a host in the remote subnet.

That certainly wasn't hard. You can also define more connections in the same configuration file. They are differentiated by the leftid and rightid parameters, allowing you to use a single configuration across many hosts.

Why This Is Difficult

First of all, if you are using iptables to filter or NAT packets, the above test may not have been successful.

The quick and easy way around NAT is to modify your MASQUERADE rule, on both ends, to exclude the internal destination on the opposite.

Traffic destined for the subnet behind the other NAT cannot be NAT'd, as NAT attempts to modify layer 4 headers, which are encrypted.

Everything in the IP data section, including TCP and UDP packets in their entirety, are encrypted and cannot be changed.

Furthermore, if you are attempting to start an IPSEC connection from a box that lives behind (a possibly secret) NAT device, it will fail. The NAT device can be configured to allow IPSEC passthrough for you, though.

Filtering traffic with IPSEC in the mix gets messy. When the packet arrives, everything but the first part of the IP header is encrypted.

Filter rules designed to detect TCP/UDP ports, or even specific upper layer protocols, will not be able to see that data until the packet is decrypted.

You can mark packets, or use policy match rules; both are convoluted, but have been discussed at length on various mailing lists including the Openswan list.

One other reason for IPSEC failing to start could be blocked ports. First get it working without iptables in the way, to make sure IPSEC is configured and working, then re-enable the firewall. Port 500 needs to be reachable by both IPSEC endpoints, for IKE, the key exchange protocol.

We can't stress enough that configuring IPSEC is actually quite painless--until you add NAT or firewalls into the picture.

You absolutely must test the IPSEC connection outside of any firewall or NAT presence to ensure it is working, before adding back the challenging piece (if you absolutely must). Your sanity will be preserved, and the installation process time dramatically reduced.