Saturday, July 9, 2016

Setup ATA over Ethernet (AoE) on Debian 8 (Initiator and Target)

https://www.howtoforge.com/tutorial/ata-over-ethernet-aoe-on-debian-8-jessie

This tutorial shows you how to setup an AoE client (initiator) and server (target) on Debian 8 /Jessie). The term AoE stands for "ATA over Ethernet" which is a storage area network (SAN) protocol that allows AoE clients to use storage devices on the (remote) AoE server over a normal ethernet network. "Remote" in this case means "inside the same LAN" because AoE is not routable outside a LAN (this is a major difference compared to iSCSI). To the AoE client (initiator), the remote storage looks like a normal, locally-attached hard drive.

1 Preliminary Note

I'm using two Debian 8 servers here:
  • server1.example.com (Initiator): IP address 192.168.1.100
  • server2.example.com (Target): IP address 192.168.1.101

2 Load the AoE Kernel Module on both systems

server1/server2:
As a first step, we have to ensure that the Kernel on our servers supports ATA Over Ethernet. Run the following command as root user.
grep ATA_OVER /boot/config-`uname -r`
This should display something like this:
root@server1:/tmp# grep ATA_OVER /boot/config-`uname -r`
CONFIG_ATA_OVER_ETH=m
root@server1:/tmp#
This means that AoE was built in as a kernel module. Now we will check if the module is loaded:
lsmod | grep aoe
If you get nothing back, this means it's not loaded. In this case, we can load it as follows:
modprobe aoe
Let's check again if the module is loaded:
lsmod | grep aoe
root@server1:/tmp# lsmod | grep aoe
aoe 51917 0
root@server1:/tmp#
To have the module loaded automatically when the system boots, we add the aoe module to /etc/modules:
nano /etc/modules
# /etc/modules: kernel modules to load at boot time.
#
# This file contains the names of kernel modules that should be loaded
# at boot time, one per line. Lines beginning with "#" are ignored.
# Parameters can be specified after the module name.

aoe

3 Setting up the Target (server2)

server2:
First, we set up the AoE target (server2):
apt-get install vblade
We can use unused logical volumes, image files, hard drives (e.g. /dev/sdb), hard drive partitions (e.g. /dev/sdb1) or RAID devices (e.g. /dev/md0) for the storage. In this example, I will use an image file of 20GB that is stored in the folder /storage.
mkdir /storage
dd if=/dev/zero of=/storage/storage1.img bs=1024k count=20000
This creates the image file /storage/storage1.img with a size of 20GB.
If you want to use logical volume instead, then you can create one with a size of 20GB named storage1 in the volume group vg0 like this:
lvcreate -L20G -n storage1 vg0
Now we export our storage device as follows:
vbladed 0 1 eth0 /storage/storage1.img
The first number (0) is the shelf number (major), the second (1) the slot number (minor), change these numbers to your liking. Each AoE device is identified by a couple major/minor which must be unique (if you are exporting multiple devices), with major between 0-65535 and minor between 0-255. The eth0 part tells vbladed which ethernet device to use (if you ethernet device is eth1, then use eth1 - you can find out about your ethernet devices by running
ifconfig
).
To start the export automatically whenever you boot the target, open /etc/rc.local...
nano /etc/rc.local
... and add the following line to it (before the exit 0 line):
[...]
/usr/sbin/vbladed 0 1 eth0 /storage/storage1.img
[...]

4 Setting up the AoE Client / Initiator (server1)

server1:
On server1, we install the initiator:
apt-get install aoetools
Now we check what AoE storage devices are available:
aoe-discover
Don't worry, the command will not show any output. The command:
aoe-stat
should now show the storage devices:
root@server1:/tmp# aoe-stat
e0.1 20.971GB eth0 1024 up
root@server1:/tmp#
At this point we have a new block device available on the client box named /dev/etherd/e0.1. If we have a look at the /dev tree a new node appears:
ls -la /dev/etherd/
root@server1:/tmp# ls -la /dev/etherd/
total 0
drwxr-xr-x 2 root root 160 Mar 22 08:46 .
drwxr-xr-x 19 root root 3160 Mar 22 08:34 ..
c-w--w---- 1 root disk 152, 3 Mar 22 08:34 discover
brw-rw---- 1 root disk 152, 0 Mar 22 08:46 e0.1
cr--r----- 1 root disk 152, 2 Mar 22 08:34 err
c-w--w---- 1 root disk 152, 6 Mar 22 08:34 flush
c-w--w---- 1 root disk 152, 4 Mar 22 08:34 interfaces
c-w--w---- 1 root disk 152, 5 Mar 22 08:34 revalidate
root@server1:/tmp#
To use that /dev/etherd/e0.1 device, we must format it:
fdisk /dev/etherd/e0.1
root@server1:/tmp# fdisk /dev/etherd/e0.1
Welcome to fdisk (util-linux 2.25.2).
Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.
Device does not contain a recognized partition table.
Created a new DOS disklabel with disk identifier 0x2922f0be.
Command (m for help): <-- n="" span="">
Partition type
p primary (0 primary, 0 extended, 4 free)
e extended (container for logical partitions)
Select (default p): <-- p="" span="">
Partition number (1-4, default 1): <-- 1="" span="">
First sector (2048-40959999, default 2048): <-- enter="" span="">
Last sector, +sectors or +size{K,M,G,T,P} (2048-40959999, default 40959999): <-- enter="" span="">
Created a new partition 1 of type 'Linux' and of size 19.5 GiB.
Command (m for help): <-- span="" w="">
The partition table has been altered.
Calling ioctl() to re-read partition table.
Syncing disks.
Now we create a filesystem on /dev/etherd/e0.1p1...
mkfs.ext4 /dev/etherd/e0.1p1
root@server1:/tmp# mkfs.ext4 /dev/etherd/e0.1p1
mke2fs 1.42.12 (29-Aug-2014)
Creating filesystem with 5119744 4k blocks and 1281120 inodes
Filesystem UUID: 2342cd83-bd45-4975-96c0-b0f366b73778
Superblock backups stored on blocks:
32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
4096000
Allocating group tables: done
Writing inode tables: done
Creating journal (32768 blocks): done
Writing superblocks and filesystem accounting information: done
... and mount it for test purposes:
mount /dev/etherd/e0.1p1 /mnt
You should now see the new device in the outputs of...
mount
root@server1:/tmp# mount
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
udev on /dev type devtmpfs (rw,relatime,size=10240k,nr_inodes=125556,mode=755)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
tmpfs on /run type tmpfs (rw,nosuid,relatime,size=204220k,mode=755)
/dev/sda1 on / type ext4 (rw,relatime,errors=remount-ro,data=ordered)
securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev)
tmpfs on /run/lock type tmpfs (rw,nosuid,nodev,noexec,relatime,size=5120k)
tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,mode=755)
cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/lib/systemd/systemd-cgroups-agent,name=systemd)
pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime)
cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset)
cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpu,cpuacct)
cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices)
cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer)
cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (rw,nosuid,nodev,noexec,relatime,net_cls,net_prio)
cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio)
cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event)
systemd-1 on /proc/sys/fs/binfmt_misc type autofs (rw,relatime,fd=23,pgrp=1,timeout=300,minproto=5,maxproto=5,direct)
mqueue on /dev/mqueue type mqueue (rw,relatime)
debugfs on /sys/kernel/debug type debugfs (rw,relatime)
hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime)
rpc_pipefs on /run/rpc_pipefs type rpc_pipefs (rw,relatime)
/dev/etherd/e0.1p1 on /mnt type ext4 (rw,relatime,data=ordered)
root@server1:/tmp#
... and
df -h
root@server1:/tmp# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/sda1 57G 1.1G 54G 2% /
udev 10M 0 10M 0% /dev
tmpfs 200M 4.6M 195M 3% /run
tmpfs 499M 0 499M 0% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs 499M 0 499M 0% /sys/fs/cgroup
/dev/etherd/e0.1p1 20G 44M 19G 1% /mnt
root@server1:/tmp#
You can unmount it like this:
umount /mnt
To have the device mounted automatically at boot time, e.g. in the directory /storage, we create that directory...
mkdir /storage
... and add the following line to /etc/fstab:
nano /etc/fstab
[...]
/dev/etherd/e0.1p1       /storage        ext4    defaults,auto,_netdev 0 0
This alone isn't enough to have the device mounted at boot time because the AoE stuff gets loaded after /etc/fstab is read. Therefore we open /etc/rc.local...
nano /etc/rc.local
... and add the following lines to it (before the exit 0 line):
[...]
aoe-discover
sleep 5
mount -a
[...]
For test purposes, you can now reboot the system:
reboot
After the reboot, the device should be mounted:
mount
root@server1:/tmp# mount
[...snip...]
/dev/etherd/e0.1p1 on /storage type ext4 (rw,relatime,data=ordered)
[...snip...]
df -h
root@server1:/tmp# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/sda1 57G 1.1G 54G 2% /
udev 10M 0 10M 0% /dev
tmpfs 200M 4.6M 195M 3% /run
tmpfs 499M 0 499M 0% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs 499M 0 499M 0% /sys/fs/cgroup
/dev/etherd/e0.1p1 20G 44M 19G 1% /storage
Now we can do a test write on the mounted partition:
touch /storage/test.txt
The check with ls command if the file has been written:
ls -la /storage
root@server1:/tmp# ls -la /storage
total 24
drwxr-xr-x 3 root root 4096 Mar 22 09:06 .
drwxr-xr-x 23 root root 4096 Mar 22 09:05 ..
drwx------ 2 root root 16384 Mar 22 09:00 lost+found
-rw-r--r-- 1 root root 0 Mar 22 09:06 test.txt
root@server1:/tmp#
The test.txt file has been written sucessfully on the volume that we mounted from server2.

5 Links

Posted by at

No comments:

Post a Comment

Subscribe to: Post Comments (Atom)